Benchmark
We benchmarked every detector we ship, honestly — including where the simple ones beat us
Every number on this page comes from labeled synthetic fleets replayed through the same production engine and the same warning-tier control charts the cloud product runs, at their shipped defaults, from frozen seeds anyone can re-run. The classical detectors usually catch faster; the budgeted tier is roughly two orders of magnitude quieter on the same healthy streams. That trade — and exactly where each side wins — is the whole story below.
Quiet vs speed — the whole suite on one fleet
200 healthy synthetic streams of 90 points (AR(1) wander plus heavy-tail spikes — no breaks anywhere), then 240 break cells on the same texture: six break shapes × 40 seeds. The series carry no declared cadence, so lags are stated in points. Every alarm on the healthy fleet is, by construction, false.
| Detector (exact rule) | Tier | False alarms on 18,000 healthy points200 synthetic streams × 90 pts, nothing breaks | Breaks caught6 break shapes × 40 seeds | Median catch lag | Pure steps caughtwhere sensitivity honestly wins |
|---|---|---|---|---|---|
| e-detector (Shiryaev–Roberts, ARL 2000)SR e-detector on conformal misses, ARL target 2000, null mean 0.1 | PAGE | 2 · 1/200 streams | 127/240 | +22 pts | 24/80 |
| coverage e-process (δ = 0.01)anytime-valid coverage e-process, miss target 0.1, false-alarm budget δ = 0.01 per stream | PAGE | 0 · 0/200 streams | 33/240 | +26.8 pts | 2/80 |
| CUSUM (k = 0.5, ARL 2000)two-sided CUSUM on standardized residuals, k = 0.5, h solved for iid-Gaussian ARL 2000, train 30 | WARN | 756 · 177/200 streams | 239/240 | +5.8 pts | 80/80 |
| EWMA chart (λ = 0.2, 3σ)EWMA λ = 0.2 with exact time-varying 3σ limits, train 30 | WARN | 465 · 180/200 streams | 208/240 | +8 pts | 70/80 |
| Static threshold (mean ± 3σ)mean ± 3σ frozen from the first 30 points, upcrossing events | WARN | 396 · 144/200 streams | 224/240 | +8 pts | 74/80 |
| Rolling band (trailing-30, ± 3σ)trailing-30 mean ± 3σ re-fit every point, upcrossing events | WARN | 248 · 167/200 streams | 174/240 | +7.5 pts | 49/80 |
| The two-tier suite — classical warnings + budgeted pagescounted caught when the page tier or the CUSUM warning tier fires | WARNPAGE | 2 budgeted pages | 239/240 | warning first, page confirms | 80/80 |
The two sentences that matter: on 18,000 healthy synthetic points, the budgeted page tier produced 2 false pages (its stated budget allowed ≈9) while the ARL-2000-tuned CUSUM produced 756 false warnings on the very same streams. And on the same texture the CUSUM caught breaks at a median of +5.8 points against the page tier’s +22 — the classical chart is genuinely faster, which is exactly the sensitivity its false warnings come from. We sell the quiet, not the sprint; the suite ships both tiers so you get the early hint and the alarm you can stake a pager on.
What each detector promises, vs what actually happens
Long healthy streams (100 streams × 2,000 points, nothing breaks), two worlds: the iid-Gaussian world the classical calibrations are priced for, and a realistic wander-plus-spikes texture. Alarm spacing is total detect-eligible points divided by total events.
| Detector | Tier | Nominal false-alarm spacing | Observed — iid Gaussianthe model world | Observed — wander + spikesthe realistic texture |
|---|---|---|---|---|
| CUSUM (k = 0.5, ARL 2000) | WARN | 1 per 2,000 pts | 1 per 189 pts | 1 per 13 pts |
| EWMA chart (λ = 0.2, 3σ) | WARN | ≈1 per 560 pts | 1 per 118 pts | 1 per 26 pts |
| Static threshold (mean ± 3σ) | WARN | ≈1 per 370 pts | 1 per 148 pts | 1 per 27 pts |
| Rolling band (trailing-30, ± 3σ) | WARN | ≈1 per 370 pts | 1 per 140 pts | 1 per 52 pts |
| e-detector (Shiryaev–Roberts, ARL 2000) | PAGE | ≤1 per 2,000 pts | none observed | 1 per 663 pts |
| coverage e-process (δ = 0.01) | PAGE | ≤1% of streams, ever | 0/100 streams fired | 0/100 streams fired |
The classical shortfall decomposes cleanly, and none of it is exotic: the ARL-2000 threshold is solved for a one-sided chart with known mean and sigma; running it two-sided (the shipped default — regressions go both ways) halves the spacing, and standardizing on the 30 training points practitioners actually have costs roughly another 4×. Multiply those and you get the observed ≈1-per-189 spacing before the data misbehaves at all. Add realistic autocorrelation and heavy tails and it collapses to ≈1 per 13 points — about 154× the nominal false-warning rate. Model-based calibration evaporates off-model; that is a property of the math, not of any vendor’s implementation.
The page tier, reported with the same honesty: the coverage e-process held its budget exactly — 0 of 100 streams fired in both worlds, against a stated 1-in-100 cap. The e-detector logged no events at all in the iid world, and on the textured 2,000-point streams its observed spacing was 1 per 663 points against its 2,000-point target: its run-length promise is conditional on the conformal miss probability staying at or below its null of 0.1, and this texture’s fastest excursions transiently push genuine miss probability above that null — the precise condition it exists to flag. On the 90-point fleets that match the product’s monitoring horizon, it stayed well inside budget: 2 events across 18,000 points, where the target allows ≈9.
The break matrix — and the concession, up front
Six break shapes, 40 seeds each, on two synthetic textures. The onset lands after 60 normal points; a detector catches a cell if it fires after the onset, and anything it fires before the onset is counted against it.
| Break shapetexture: wander + heavy-tail spikes · onset after 60 normal pts | CUSUM warning (lag) | Page tier (lag) | Suite caught | Warning → page confirmmedian gap when both fire |
|---|---|---|---|---|
| slow drift (+0.5σ₀/8 pts) | 39/40 +6 pts | 3/40 +27 pts | 39/40 | +20 pts (n=3) |
| steady drift (+1.0/pt) | 40/40 +6.5 pts | 21/40 +27 pts | 40/40 | +17 pts (n=21) |
| fast drift (+1.6/pt) | 40/40 +8 pts | 39/40 +24 pts | 40/40 | +15 pts (n=39) |
| step +10 then creep +1.2/pt | 40/40 +5 pts | 40/40 +20 pts | 40/40 | +14 pts (n=40) |
| pure step +20 (≈3σ of texture) | 40/40 +3 pts | 19/40 +19 pts | 40/40 | +13 pts (n=19) |
| pure step +12 (≈2σ of texture) | 40/40 +5.5 pts | 5/40 +15 pts | 40/40 | +13 pts (n=5) |
| Break shapetexture: smooth near-pink · onset after 60 normal pts | CUSUM warning (lag) | Page tier (lag) | Suite caught | Warning → page confirmmedian gap when both fire |
|---|---|---|---|---|
| slow drift (+0.5σ₀/8 pts) | 40/40 +5.5 pts | 12/40 +23.5 pts | 40/40 | +19.5 pts (n=12) |
| steady drift (+1.0/pt) | 40/40 +6 pts | 34/40 +23.5 pts | 40/40 | +15 pts (n=34) |
| fast drift (+1.6/pt) | 40/40 +5 pts | 40/40 +19.5 pts | 40/40 | +13 pts (n=40) |
| step +10 then creep +1.2/pt | 40/40 +3.5 pts | 40/40 +15 pts | 40/40 | +12 pts (n=40) |
| pure step +20 (≈3σ of texture) | 40/40 +2 pts | 23/40 +16 pts | 40/40 | +14 pts (n=23) |
| pure step +12 (≈2σ of texture) | 40/40 +3 pts | 12/40 +15.5 pts | 40/40 | +14 pts (n=12) |
The concession first: on pure steps the e-process tier alone is the wrong tool — it missed most of them, while the CUSUM warning tier caught every single one within a handful of points. That is why the product ships the suite rather than the e-process alone: across all 480 break cells the two-tier suite caught 479 (479/480), against 288 for the page tier by itself. The lifecycle the numbers support: the warning fires early and unbudgeted, and when the incident is real the budgeted page confirms it a median of 12–20 points later — and when it is not, the warning quietly expires instead of waking anyone.
Methodology
The harness regenerates every number on this page with one command:
cd backend && uv run python scripts/bench/run_bench.pyA standalone benchmark repository — github.com/validanytime/bench — publishes with the launch. It ships the frozen series generators and seeds, the full machine-readable results, and a pure-NumPy replayer for the four warning-tier detectors (a line-for-line port of the product implementations, golden-tested against their outputs), so every warning-tier row can be re-verified without the engine. The page-tier rows come from the production engine, which is not public: in that repository they ship as frozen results checked against the published JSON rather than regenerated.
Data
- All series are synthetic and labeled as such. Four generators: iid Gaussian (σ = 6); AR(1) wander (ρ = 0.9, σₑ = 2.6) plus iid Gaussian noise; AR(1) wander plus t(3) spikes (the “realistic texture” used for the headline table); and a slow ρ = 0.97 wander with long excursions. Seeds are frozen constants in
backend/scripts/bench/bench_lib.py— same code, same bytes out. - Break shapes: three drift slopes, a step-plus-creep, and two pure steps, all injected at point 60 of 90.
Detectors
- Every row is the product implementation at its shipped defaults — not a reimplementation: the vendored engine’s conformal calibrator (window 64, α = 0.1) feeding the Shiryaev–Roberts (SR) e-detector (ARL target 2,000) and the coverage e-process (δ = 0.01), plus the four warning-tier control charts exactly as the cloud product runs them. The baseline each stream monitors against is the product’s zero-config law: the median of the first 30 points, frozen.
- Event semantics follow each chart’s own convention: CUSUM resets after an alarm (renewal), EWMA / static / rolling-band alarm on upcrossing events, and the e-process monitors are counted on upcrossings of their alarm level.
- The benchmark grades shipped defaults. A statistician hand-tuning any of these charts per-stream would do better than its row here; almost nobody operates fleets that way, which is the point of benchmarking defaults.
What we do not claim
- Synthetic fleets are not your data. They are constructed to be fair-but-hard (wander, spikes, slow slides) and every generator parameter is published; run the same suite on your own history in the browser before believing any table, ours included.
- No claim that the e-process is faster — the tables above show it is usually slower. The claim under test is that its false-alarm budget survives realistic data, and the classical calibrations do not.
- Warning-tier detectors carry no guarantee here or anywhere: their calibration is model-based and the calibration table shows what that means off-model. Budget language belongs to the page tier only.
Downloads
- validanytime-bench-v1.json — every study, machine-readable, with seeds and configs embedded.
Run this bake-off on your own history
The same detectors, in your browser, on your pasted or uploaded metric history — nothing is uploaded, and the verdict is the production engine’s. Or read how each detector works, when it wins, and when it lies.