Documentation menu
Concepts
Anytime-valid 101
Anytime-valid inference is statistics whose guarantees survive continuous looking: e-values, e-processes, and confidence sequences let you check a stream as often as you like — and stop whenever you like — without inflating error rates.
The peeking problem
A classical test controls error for one look at a fixed sample size. Check the same metric repeatedly and you are running many tests — and the probability that at least one trips by chance climbs fast. After ~20 looks at a fixed threshold, the false-positive rate approaches ~64%. This has been known since Robbins (1952) and Armitage; it is why naive dashboards both cry wolf and miss real failures.
E-values and e-processes
An e-value is a measure of evidence against a null hypothesis with a simple, powerful property: under the null its expected value is at most one. An e-process is a sequence of e-values you can monitor over time. Because of that bounded-expectation property, you can stop and look whenever you want — the false-alarm guarantee still holds. This is what “anytime-valid” means. The underlying result is Ville’s inequality (Ville, 1939); for the modern framework, see Ramdas, Grünwald, Vovk & Shafer (2023).
Confidence sequences
A confidence sequence is the interval counterpart: a band that covers the true value simultaneously across all time, not just at one pre-chosen sample size. You can watch it continuously and act on it without inflating error. The construction follows Howard, Ramdas, McAuliffe & Sekhon (2021).
ARL-calibrated e-detectors
For fast change detection we use e-detectors (Shiryaev–Roberts — SR — / CUSUM-style), calibrated to a target Average Run Length (ARL) — the expected time between false alarms when nothing is wrong. Pick an ARL (say 2,000 steps) and the detector’s threshold is set so it stays quiet on stable data and reacts quickly when a real change begins. The e-detector framework is due to Shin, Ramdas & Rinaldo (2024); the SR statistic itself goes back to the Shiryaev–Roberts procedure (Shiryaev, 1963; Roberts, 1966).
Online false-discovery control
Watching many streams at once reintroduces multiplicity. ValidAnytime applies online FDR procedures (e-LOND / e-LORD; see Xu & Ramdas, 2024) that bound the false-discovery rate across the whole fleet as monitors fire over time — not one threshold in isolation.
Two alarm tiers: warnings and pages
ValidAnytime also runs the classical control charts — CUSUM, EWMA, static thresholds, rolling bands — as a warning tier alongside the e-process page tier. The classical detectors are sensitive and fast, but their false-warning calibration is model-based (it assumes iid standardized inputs), so on realistic data — autocorrelated wander, heavy-tailed spikes — their false-warning rate can exceed the nominal rate by orders of magnitude; the benchmark measures that gap on labeled synthetic streams. That is not a reason to discard them; it is a reason to label them honestly: warnings arrive early and unbudgeted, and the page — the alarm you wake someone for — is reserved for the e-process tier, whose false-alarm control does not depend on the model being right.
The receipt: guarantee_tag and theorem_ref
Every alarm carries a guarantee_tag (which guarantee applies — e.g. average_run_length_e_detector for the page tier, or heuristic_adaptive for a warning-tier alarm that carries no anytime-valid guarantee at all) and a theorem_ref (the result that backs it). You always know exactly what was promised and under what assumptions.
Rigor is the engine; outcomes are the packaging. You ship the decision — the math is there when you need to defend it.
Ready to use it? Head to the Quickstart.