Benchmarking drift detectors, honestly

We sell the budgeted tier, so you should expect this write-up to flatter it — which is exactly why the benchmark is built so it can’t. Every detector we ship, classical charts included, was replayed against the same labeled synthetic fleets through the production engine, from frozen seeds, and the full grid is published — including the cells where the classical charts beat us.

What was measured

The headline fleet is 200 synthetic streams of 90 points each — 18,000 points with no break anywhere — on a realistic texture: slow AR(1) wander plus heavy-tailed spikes, the shape production latency actually has. Alongside it, a break matrix injects 480 labeled breaks (two textures × six break shapes × 40 seeds) so catch rates and lags are measured on breaks we know are real. Everything is synthetic and labeled as such; that is the price of knowing the ground truth, and the benchmark page spells out what synthetic can and cannot show.

The healthy fleet: who cries wolf

On those 18,000 healthy points, the two-sided CUSUM at the product’s default parameters (k = 0.5, threshold solved for an iid-Gaussian ARL of 2,000) raised 756 false warnings across 177 of 200 streams. The budgeted page tier raised 2 false pages against a stated budget of ≈9. That is the product’s two-tier split in one line: the classical chart is a sensitive early-warning system with no budget attached, and the page tier is the alarm whose false-alarm budget actually holds.

Where CUSUM honestly wins

On sudden steps, CUSUM caught 80 of 80 injected breaks on this texture with a median lag around 5.75 points — genuinely faster than the e-detector, which caught 24 of 80 steps at a median lag of 22 points. Across the full matrix the budgeted tier alone caught 288 of 480 breaks — slower and blinder than the classical charts, which is the honest cost of carrying a guarantee. If someone tells you an anytime-valid detector dominates CUSUM on speed, they are selling something; the benchmark says otherwise, per shape, in public.

Why the suite ships both

Run together — classical charts as unbudgeted warnings, the page tier as the budgeted alarm — the suite caught 479 of 480 injected breaks while keeping false pages inside the stated budget on the healthy fleet. You get CUSUM’s speed where it is real and a page you can trust when it fires. The catch is that CUSUM’s own calibration promise does not survive contact with realistic data: tuned for one false alarm per 2,000 points, it delivered one per ~189 on textbook-iid streams and one per ~13 on the realistic texture. That decomposition — none of it exotic, all of it measurable — gets its own write-up.

The full grid — every detector, every break shape, every lag, plus the calibration table and a downloadable results file — is at /bench, and the detector guides let you run each rule on synthetic data in your browser.