The peeking problem is the reason a metric can look 'significant' just because you checked it too many times.
Also known as: optional stopping / repeated significance testing
The peeking problem is the trap that catches almost everyone: you have a test, you check it repeatedly as data comes in, and you stop the moment it looks bad. Even if nothing is actually wrong, keep looking long enough and pure noise will eventually cross any fixed line. Every extra look is another roll of the dice, and classical tests were never designed to be checked more than once.
It is exactly why monitoring dashboards cry wolf. A threshold that would false-alarm 5% of the time if you looked once will false-alarm far more often if you look every minute. Teams respond by muting alerts — and then miss the real regression.
Anytime-valid methods solve the peeking problem at the root: their guarantees are proven to hold across all looks at once, so continuous checking is not just allowed, it is the intended use.
ValidAnytime turns these ideas into a live alarm you can trust — valid no matter how often you look. Prove it on your own data, free.