ValidAnytime
  • Try it free
  • Observatory
  • Pricing
  • Docs
Sign inStart free
ValidAnytime

A guaranteed false-alarm budget across your whole fleet, valid no matter how often you look.

Made by Compiled Intelligence — a frontier AI lab working on quantitative finance from first principles; ValidAnytime is the monitoring we built for our own model fleets, productized.

Product

  • How it works
  • Pricing
  • Observatory
  • Try it free
  • Dashboard

Solutions

  • Who it's for
  • For LLM engineers
  • For MLOps engineers
  • For SRE & platform
  • LLM eval regression
  • Model drift detection
  • Alert fatigue

Developers

  • Docs
  • Quickstart
  • Python SDK
  • API reference

Learn

  • Benchmark
  • Detector guides
  • Glossary
  • Compare
  • Why dashboards lie
  • Blog
  • Anytime-valid 101
  • Winter Storm Uri case study
  • Incident ledger
  • Observatory methodology

Company

  • Contact
  • Privacy
  • Terms

Observatory alarms & product updates, by email

Rare emails only. We store your address and the tag “footer” — nothing else. Unsubscribe anytime.

© 2026 ValidAnytime. All rights reserved.

Every alarm ships with its guarantee_tag and theorem_ref.

Glossary

Always-valid p-value

An always-valid p-value is a p-value you are allowed to read at any moment — it never gets less trustworthy the more often you check.

Also known as: anytime-valid p-value

An always-valid p-value is a p-value defined for every moment at once, so you can watch it fall as data arrives and act the instant it crosses your line — no penalty for looking. A classical p-value comes with fine print: it is only honest if you decided the sample size in advance and looked exactly once. Peek early, peek often, or stop when it dips below 0.05, and the number quietly lies. An always-valid p-value removes the fine print.

For monitoring, this is what lets a threshold actually mean what it says. Set the alarm at, say, 0.01 and the guarantee holds however many times you refresh the page, because the validity was built to survive continuous checking.

Mechanically, an always-valid p-value is just the reciprocal of a running e-process, capped at one. That is why the same object powers both an alarm (the e-value) and a familiar p-value — two views of the same anytime-valid evidence.

Go deeper

  • Anytime-valid 101 in the docs
  • Why your dashboard is lying to you

Related terms

  • E-processAn e-process is a running score of evidence against 'nothing has changed'; its value at any moment is an e-value, and it stays valid at every look.
  • Anytime-valid inferenceAnytime-valid inference is a way of testing that stays statistically valid no matter how often you look at the results.
  • The peeking problemThe peeking problem is the reason a metric can look 'significant' just because you checked it too many times.
  • Ville's inequalityVille's inequality is the theorem that makes 'valid no matter how often you look' true rather than wishful.

Put the theory to work.

ValidAnytime turns these ideas into a live alarm you can trust — valid no matter how often you look. Prove it on your own data, free.

Start freeTry the detector