# ValidAnytime

> A guaranteed false-alarm budget across your whole fleet, valid no matter how often you look.

ValidAnytime is anytime-valid monitoring for ML models, LLM systems, and any
metric that matters. Fixed-threshold dashboards false-alarm the more often you
look — check a live metric long enough and noise will eventually trip it. We
use anytime-valid statistics (e-processes, confidence sequences, ARL-calibrated
e-detectors) so the false-alarm guarantee holds no matter how often you look,
with online false-discovery control bounding false alarms across your whole
fleet. Every alarm ships a guarantee tag and theorem reference.

Monitors run a TWO-TIER alarm policy. The warning tier is the classical
detector suite (cusum, ewma_chart, static_threshold, rolling_band) — sensitive
and fast, but calibrated against an iid model, so warnings arrive unbudgeted
and their alarms carry guarantee_tag "heuristic_adaptive" with e_value null.
The page tier (coverage_e_process, e_detector, risk_e_process) delivers alarms
inside a stated false-alarm budget; only page-tier e-values enter the fleet
FDR pool. Templates (llm_quality, ml_drift, kpi) resolve server-side into a
complete two-tier config at create; the resolved config is persisted and
returned. A monitor's baseline (mu_baseline) can be set explicitly or is
learned from the first 30 values and then frozen. Invalid configs are 422 at
create, never a 500 at first ingest.

## Architecture (important for accuracy)

- The anytime-valid ENGINE (detectors, fleet-wide FDR, backtest gate) runs in
  the ValidAnytime CLOUD. There is no self-hosted engine today.
- The `validanytime` Python SDK is a THIN, OPEN-SOURCE (MIT) HTTP client
  for the hosted API. Using it means streaming metric values to the API and
  reading alarms back; it does not run the detector locally.
- What runs entirely in your browser: the /try backtest and the /detectors
  playgrounds, both built on a line-for-line TypeScript port of the cloud
  engine's backtest path — same conformal calibrator, same e-process monitors,
  same alarm thresholds — verified against the production engine's own outputs
  by golden tests. Nothing pasted or uploaded at /try leaves the browser.
- Status: pre-launch. Sample and illustrative data are labeled as such.

## Key pages

- [Home](https://validanytime.com/): the problem (alert fatigue, regressions that sit green) and the fix.
- [Docs](https://validanytime.com/docs): overview of how ValidAnytime turns a metric stream into trustworthy alarms.
- [Quickstart](https://validanytime.com/docs/quickstart): create a monitor and stream values with the SDK; point an assistant at the API.
- [Python SDK](https://validanytime.com/docs/sdk): the thin, typed, MIT HTTP client — methods, idempotency, return types.
- [API reference](https://validanytime.com/docs/api): the REST API — monitors, idempotent ingest, alarms, and the backtest gate.
- [Anytime-valid 101](https://validanytime.com/docs/concepts): the peeking problem, e-values, confidence sequences, online FDR.
- [Observatory](https://validanytime.com/observatory): the same engine, live on a REAL deployed model — each US grid's day-ahead electricity-demand forecast (EIA-930). We monitor error = actual − forecast and page only when it changes regime, inside a stated false-alarm budget. A live demo of production model-monitoring; the grid's forecast stands in for yours.
- [Incident ledger](https://validanytime.com/observatory/ledger): every dated forecast-error regime change since 2021, permanent and budgeted, cross-referenced to documented grid-stress events (Winter Storm Uri, Elliott, the 2026 cold wave). Subscribe by RSS.
- [Winter Storm Uri case study](https://validanytime.com/observatory/story/uri): what anytime-valid monitoring sees when a model's world comes apart — ERCOT's actual demand fell ~30 GW below its own forecast as load was shed, and the budgeted alarm still fires once.
- [The wall](https://validanytime.com/wall): two alarm policies side by side on one labeled synthetic fleet — a static 3σ threshold's false pages vs the budgeted page tier's, minute by minute.
- [Observatory methodology](https://validanytime.com/observatory/methodology): the full recipe — error = actual − day-ahead forecast, conformal band, ARL-calibrated e-detector, per-grid adaptive calibration, and what we do NOT claim (a forecast-error regime change, never grid failure).
- [Grid status badges](https://validanytime.com/observatory/badge/<ba>): an embeddable, self-updating SVG status pill for any monitored grid — e.g. https://validanytime.com/observatory/badge/ERCO — shields.io-style, reading "no regime change" while the forecast holds or "forecast drift detected" when the e-detector pages, and linking back to that grid's verdict page. Copy-paste Markdown/HTML is on each grid's page; embed it in a README, status page, or dashboard.
- [Compare](https://validanytime.com/compare): honest "vs" pages against dashboards, APM, ML and LLM monitoring tools.
- [Solutions](https://validanytime.com/use-cases): landing pages by problem and by team.
- [For LLM engineers](https://validanytime.com/for/llm-engineers): catch silent model updates and eval regressions the day the evidence becomes real.
- [For MLOps engineers](https://validanytime.com/for/mlops-engineers): drift alarms that don't cry wolf — proven on your own history first.
- [For SRE & platform teams](https://validanytime.com/for/sre-platform): one trustworthy page per real incident, with a bounded false-alarm budget.
- [LLM eval regression detection](https://validanytime.com/use-cases/llm-eval-regression): one trustworthy alarm per real drop in judge scores or task success.
- [Model drift detection](https://validanytime.com/use-cases/model-drift-detection): detect data and concept drift without the daily PSI/KS false alarms.
- [Alert fatigue](https://validanytime.com/use-cases/alert-fatigue): why fixed thresholds must false-alarm, and the anytime-valid fix.
- [Glossary](https://validanytime.com/glossary): plain-English definitions of the vocabulary behind the method.
- [Try a backtest](https://validanytime.com/try): replay your own metric history through a line-for-line port of the cloud engine's backtest path (golden-tested against production) — paste or upload a CSV, in your browser, no signup; results are shareable as summary cards. Every result also shows the bake-off: how the four classical detectors (static threshold, rolling band, EWMA, CUSUM) would have done on the same history, false alarms and catch lags included, next to the engine.
- [Benchmark](https://validanytime.com/bench): reproducible detector benchmark — CUSUM, EWMA, static threshold, and rolling band vs the anytime-valid e-process tier on labeled synthetic fleets replayed through the production engine; frozen seeds, downloadable results, and an explicit account of where the classical detectors honestly win. A standalone benchmark repository (frozen generators + a pure-NumPy warning-tier replayer) publishes with the launch.
- [Detector guides](https://validanytime.com/detectors): every detector in the suite explained honestly — exact update rules, live in-browser playgrounds, and what each does and does not guarantee (CUSUM, EWMA, static threshold, rolling band, e-detector, coverage e-process).
- [Docs for agents](https://validanytime.com/docs/agents): copy-paste prompts and MCP setup for coding agents — connect the MCP server, create a monitor from a template, and run the backtest gate for a machine-readable verdict.
- [Full reference for agents](https://validanytime.com/llms-full.txt): the expanded machine-readable surface — API endpoints, config schema, guarantee-tag vocabulary, SDK methods, MCP tools.
- [Pricing](https://validanytime.com/pricing): plans and the free tier.
- [Blog](https://validanytime.com/blog): essays on why dashboards lie and how anytime-valid inference fixes it.

## Getting started

Start free at https://validanytime.com/signup. See the quickstart for the first monitor.
To try it without an account, run a backtest on your own history at https://validanytime.com/try.