Python SDK

What’s open source

The validanytime Python package is open source (MIT): it is a thin, typed HTTP client for the ValidAnytime API. The anytime-valid engine — the detectors, fleet-wide false-discovery control, and the backtest gate — runs in the ValidAnytime cloud. Using the SDK means streaming your chosen metric values to the API and reading alarms back; the engine does not run locally, and there is no self-hosted deployment today. (Want to see the math run entirely in your browser instead? The backtest playground does exactly that on your own numbers.)

Install

pip install validanytime

Preview — SDK at launch. The package publishes to PyPI at launch — start free for install instructions.

Initialize

from validanytime import Client

va = Client(
    api_key="va_...",
    base_url="https://api.validanytime.com",  # optional
)

Core methods

# create / fetch monitors
monitor = va.create_monitor(name="llm-answer-quality",
                            config={"template": "llm_quality"})
monitor = va.get_monitor(monitor.id)

# ingest one event (event_id makes it idempotent on retry)
va.ingest(monitor.id, value=0.91, event_id="evt_1042",
          ts="2026-06-24T14:02:11Z", label=1.0)

# ingest a batch
va.ingest_batch(monitor.id, events=[
    {"value": 0.88, "event_id": "evt_1043"},
    {"value": 0.86, "event_id": "evt_1044"},
])

# list alarms (optionally since a timestamp)
alarms = va.alarms(monitor.id, since="2026-06-01T00:00:00Z")

Autoconfig & the backtest gate

Don’t hand-write detector math — bring the data and let the backtest gate (not any model) ratify a config before it goes live. This is the agent-driven path: your agent reads the metric or asks the user, then calls the API.

# Autoconfig: read the metric's shape, get a GATE-RATIFIED config.
sample = [...]                                 # your recent metric values
s = va.suggest_config(sample, history=sample)  # -> Suggestion
if s.backtest.passed:                          # the non-bypassable check
    m = va.create_monitor("my-metric", config=s.config)
    print(s.backtest.summary)                  # "...caught it on day X."

# Data-free matrix lookup, or re-run the gate on any config.
rows = va.use_cases()                          # UseCase(id, direction, config, ...)
report = va.backtest(sample, config=rows[0].config)

• suggest_config(sample, history=…, use_case=…) → Suggestion (config — resolved and gate-ratified — plus backtest). It reads the sample’s shape (level, scale, direction, boundedness, drift), tunes a config, and runs the gate on history. Always check suggestion.backtest.passed before going live.
• backtest(history, config=…, inject_shift=…) → BacktestReport. Graded on the page tier (passed, caught_at_seq); a warn-tier fire on the normal stretch is reported as warned_on_normal and never fails the gate. A backtest is a replay of past data, not a forecast about a different stream.
• use_cases() → a list of UseCase — the public matrix (decision table) as a data-free lookup: each row’s resolved config is create-ready, and its id is a valid use_case= hint.

Idempotency & ordering

Pass an event_id with every event. Ingestion is an append-only log drained in strict order, so retries are de-duplicated and the anytime-valid guarantee — which depends on event order — is preserved. Re-sending the same event_id is a safe no-op.

Validation at create

create_monitor validates the config by constructing the real engine server-side: an unknown monitor family, a bad kwarg, or an unknown template id raises a 422 with the monitor menu in the message — never a 500 on your first event. A template config resolves into the full two-tier config it runs (page-tier e-processes plus a warning-tier control chart), and the Monitor.config you get back is that resolved config.

Return types

• create_monitor / get_monitor → Monitor (id, name, config — the resolved config that actually runs — status).
• ingest / ingest_batch → IngestResult (accepted, duplicate).
• alarms → a list of Alarm (seq, fired_at, statistic, guarantee_tag, theorem_ref, e_value, discovery, message). Warning-tier alarms carry guarantee_tag="heuristic_adaptive" and always have e_value=None — they are sensitive, model-calibrated early hints, not budgeted pages.