ValidAnytime
  • Try it free
  • Observatory
  • Pricing
  • Docs
Sign inStart free
ValidAnytime

A guaranteed false-alarm budget across your whole fleet, valid no matter how often you look.

Made by Compiled Intelligence — a frontier AI lab working on quantitative finance from first principles; ValidAnytime is the monitoring we built for our own model fleets, productized.

Product

  • How it works
  • Pricing
  • Observatory
  • Try it free
  • Dashboard

Solutions

  • Who it's for
  • For LLM engineers
  • For MLOps engineers
  • For SRE & platform
  • LLM eval regression
  • Model drift detection
  • Alert fatigue

Developers

  • Docs
  • Quickstart
  • Python SDK
  • API reference

Learn

  • Benchmark
  • Detector guides
  • Glossary
  • Compare
  • Why dashboards lie
  • Blog
  • Anytime-valid 101
  • Winter Storm Uri case study
  • Incident ledger
  • Observatory methodology

Company

  • Contact
  • Privacy
  • Terms

Observatory alarms & product updates, by email

Rare emails only. We store your address and the tag “footer” — nothing else. Unsubscribe anytime.

© 2026 ValidAnytime. All rights reserved.

Every alarm ships with its guarantee_tag and theorem_ref.

Glossary

Sequential testing

Sequential testing is the practice of testing a hypothesis as data arrives, deciding to stop as soon as the evidence is conclusive.

Also known as: sequential analysis

Sequential testing is the practice of evaluating evidence continuously rather than waiting for a fixed sample and judging once. When the evidence is decisive early, you stop early; when it is inconclusive, you keep collecting. For monitoring, that is the natural shape of the problem — data never stops arriving, and you want to react as soon as, but no sooner than, the evidence warrants.

The catch is that naïve sequential testing reintroduces the peeking problem. Doing it correctly requires methods whose error guarantees survive optional stopping — which is precisely what anytime-valid tools provide.

ValidAnytime is sequential testing done right: e-processes accumulate evidence point by point, an alarm fires the moment it is justified, and the false-alarm guarantee holds at whatever moment you chose to stop.

Go deeper

  • Anytime-valid 101 in the docs
  • CUSUM — the SPRT's descendant, explained honestly

Related terms

  • Anytime-valid inferenceAnytime-valid inference is a way of testing that stays statistically valid no matter how often you look at the results.
  • The peeking problemThe peeking problem is the reason a metric can look 'significant' just because you checked it too many times.
  • E-processAn e-process is a running score of evidence against 'nothing has changed'; its value at any moment is an e-value, and it stays valid at every look.
  • Confidence sequenceA confidence sequence is a sequence of confidence intervals that stays valid at every point in time, so you can read it whenever you like.
  • Online FDR controlOnline FDR control is a way to bound the fraction of false alarms across many streams that you are testing continuously over time.
  • Test martingaleA test martingale is a running evidence score that, if nothing has changed, is not expected to grow — the honest core of an e-process.
  • Changepoint detectionChangepoint detection is the task of spotting the moment a metric's behavior genuinely shifts — as opposed to normal noise wobbling around.
  • SPC for MLSPC for ML is the practice of putting machine-learning metrics under statistical process control — control charts with explicit rules, not eyeballed dashboards.

Put the theory to work.

ValidAnytime turns these ideas into a live alarm you can trust — valid no matter how often you look. Prove it on your own data, free.

Start freeTry the detector