ValidAnytime
  • Try it free
  • Observatory
  • Pricing
  • Docs
Sign inStart free
ValidAnytime

A guaranteed false-alarm budget across your whole fleet, valid no matter how often you look.

Made by Compiled Intelligence — a frontier AI lab working on quantitative finance from first principles; ValidAnytime is the monitoring we built for our own model fleets, productized.

Product

  • How it works
  • Pricing
  • Observatory
  • Try it free
  • Dashboard

Solutions

  • Who it's for
  • For LLM engineers
  • For MLOps engineers
  • For SRE & platform
  • LLM eval regression
  • Model drift detection
  • Alert fatigue

Developers

  • Docs
  • Quickstart
  • Python SDK
  • API reference

Learn

  • Benchmark
  • Detector guides
  • Glossary
  • Compare
  • Why dashboards lie
  • Blog
  • Anytime-valid 101
  • Winter Storm Uri case study
  • Incident ledger
  • Observatory methodology

Company

  • Contact
  • Privacy
  • Terms

Observatory alarms & product updates, by email

Rare emails only. We store your address and the tag “footer” — nothing else. Unsubscribe anytime.

© 2026 ValidAnytime. All rights reserved.

Every alarm ships with its guarantee_tag and theorem_ref.

Glossary

Online FDR control

Online FDR control is a way to bound the fraction of false alarms across many streams that you are testing continuously over time.

Also known as: false discovery rate control

Online FDR control is a way of bounding false alarms across many monitors at once: it gives you a single budget for false discoveries and spends it across every stream and every look, so the overall rate of crying wolf stays bounded. Watching one metric is hard enough; a real system has hundreds. If each has its own alarm, false positives pile up fast — even a tiny per-alarm error rate becomes a flood of noise across the fleet.

This is what turns "we have monitors" into "we have alarms we trust." Instead of tuning each threshold and hoping, you get a fleet-wide guarantee: of everything that fires, only a controlled fraction are false.

The rigor comes from online multiple-testing procedures (the LORD / SAFFRON family) adapted to anytime-valid e-values. It is the piece most dashboards simply do not have — they alert per-metric, with no global bound.

Go deeper

  • Compare against ML monitoring tools
  • How the guarantee works (docs)
  • For MLOps engineers
  • Model drift detection

Related terms

  • E-processAn e-process is a running score of evidence against 'nothing has changed'; its value at any moment is an e-value, and it stays valid at every look.
  • Anytime-valid inferenceAnytime-valid inference is a way of testing that stays statistically valid no matter how often you look at the results.
  • Sequential testingSequential testing is the practice of testing a hypothesis as data arrives, deciding to stop as soon as the evidence is conclusive.
  • False discovery rateThe false discovery rate is the fraction of fired alarms that turn out to be false — the number you actually want to control across a fleet.

Put the theory to work.

ValidAnytime turns these ideas into a live alarm you can trust — valid no matter how often you look. Prove it on your own data, free.

Start freeTry the detector