Python SDK

What’s open source

The validanytime Python package is open source (Apache-2.0): it is a thin, typed HTTP client for the ValidAnytime API. The anytime-valid engine — the detectors, fleet-wide false-discovery control, and the backtest gate — runs in the ValidAnytime cloud. Using the SDK means streaming your chosen metric values to the API and reading alarms back; the engine does not run locally, and there is no self-hosted deployment today. (Want to see the math run entirely in your browser instead? The backtest playground does exactly that on your own numbers.)

Install

pip install validanytime

Live on PyPI. Start free to mint an API key and point the client at the hosted API.

Initialize

from validanytime import Client

va = Client(
    api_key="va_...",
    base_url="https://api.validanytime.com",  # optional
)

Core methods

# create / fetch monitors
monitor = va.create_monitor(name="llm-answer-quality",
                            config={"template": "llm_quality"})
monitor = va.get_monitor(monitor.id)

# ingest one event (event_id makes it idempotent on retry)
va.ingest(monitor.id, value=0.91, event_id="evt_1042",
          ts="2026-06-24T14:02:11Z", label=1.0)

# ingest a batch
va.ingest_batch(monitor.id, events=[
    {"value": 0.88, "event_id": "evt_1043"},
    {"value": 0.86, "event_id": "evt_1044"},
])

# list alarms (optionally since a timestamp)
alarms = va.alarms(monitor.id, since="2026-06-01T00:00:00Z")

Autoconfig & the backtest gate

Don’t hand-write detector math — bring the data and let the backtest gate (not any model) ratify a config before it goes live. This is the agent-driven path: your agent reads the metric or asks the user, then calls the API.

# Autoconfig: read the metric's shape, get a GATE-RATIFIED config.
sample = [...]                                 # your recent metric values
s = va.suggest_config(sample, history=sample)  # -> Suggestion
if s.backtest.passed:                          # the non-bypassable check
    m = va.create_monitor("my-metric", config=s.config)
    print(s.backtest.summary)                  # "...caught it on day X."

# Data-free matrix lookup, or re-run the gate on any config.
rows = va.use_cases()                          # UseCase(id, direction, config, ...)
report = va.backtest(sample, config=rows[0].config)

• suggest_config(sample, history=…, use_case=…) → Suggestion (config — resolved and gate-ratified — plus backtest). It reads the sample’s shape (level, scale, direction, boundedness, drift), tunes a config, and runs the gate on history. Always check suggestion.backtest.passed before going live.
• backtest(history, config=…, inject_shift=…) → BacktestReport. Graded on the page tier (passed, caught_at_seq); a warn-tier fire on the normal stretch is reported as warned_on_normal and never fails the gate. A backtest is a replay of past data, not a forecast about a different stream.
• use_cases() → a list of UseCase — the public matrix (decision table) as a data-free lookup: each row’s resolved config is create-ready, and its id is a valid use_case= hint.

Idempotency & ordering

Pass an event_id with every event. Ingestion is an append-only log drained in strict order, so retries are de-duplicated and the anytime-valid guarantee — which depends on event order — is preserved. Re-sending the same event_id is a safe no-op.

Validation at create

create_monitor validates the config by constructing the real engine server-side: an unknown monitor family, a bad kwarg, or an unknown template id raises a 422 with the monitor menu in the message — never a 500 on your first event. A template config resolves into the full two-tier config it runs (page-tier e-processes plus a warning-tier control chart), and the Monitor.config you get back is that resolved config.

Return types

• create_monitor / get_monitor → Monitor (id, name, config — the resolved config that actually runs — status).
• ingest / ingest_batch → IngestResult (accepted, duplicate).
• alarms → a list of Alarm (seq, fired_at, statistic, guarantee_tag, theorem_ref, e_value, discovery, message). Warning-tier alarms carry guarantee_tag="heuristic_adaptive" and always have e_value=None — they are sensitive, model-calibrated early hints, not budgeted pages.