Quickstart

Set up ValidAnytime in three steps: install the Python SDK, create a monitor and stream your metric values to it, then prove the config on your own history with the backtest gate — a few minutes end to end.

1. Install the SDK

shell

pip install validanytime

Preview — SDK at launch. The SDK and public API publish at launch. Start free and we’ll send your API key and install instructions.

Works today

Dashboard — create a monitor, mint an API key, watch alarms
The REST API over HTTP (curl or any client)
The in-browser backtest at /try — no signup
The MCP server, run locally from a source checkout

Publishes at launch

pip install validanytime (the Python SDK)
The GitHub Action (offline backtest gate in CI)
The MCP registry listing (one-line install)

2. Create a monitor and stream values

from validanytime import Client

va = Client()  # reads VALIDANYTIME_API_KEY (from your dashboard)

# 1. create a monitor from a template
monitor = va.create_monitor(
    name="llm-answer-quality",
    config={"template": "llm_quality"},
)

# 2. stream values as they happen (event_id makes ingest idempotent)
va.ingest(monitor.id, value=0.91, event_id="evt_1042")

# 3. read any alarms
for alarm in va.alarms(monitor.id):
    print(alarm.message, "—", alarm.guarantee_tag)

# create a monitor
curl https://api.validanytime.com/v1/monitors \
  -H "Authorization: Bearer $VA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "llm-answer-quality", "config": {"template": "llm_quality"}}'

# stream a value (event_id makes ingest idempotent)
curl https://api.validanytime.com/v1/monitors/$MONITOR_ID/events \
  -H "Authorization: Bearer $VA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"value": 0.91, "event_id": "evt_1042"}'

// A few lines of fetch — the API is plain JSON in, JSON out.
const base = "https://api.validanytime.com";
const headers = {
  Authorization: `Bearer ${process.env.VA_API_KEY}`,
  "Content-Type": "application/json",
};

const monitor = await fetch(`${base}/v1/monitors`, {
  method: "POST",
  headers,
  body: JSON.stringify({ name: "llm-answer-quality", config: { template: "llm_quality" } }),
}).then((r) => r.json());

await fetch(`${base}/v1/monitors/${monitor.id}/events`, {
  method: "POST",
  headers,
  body: JSON.stringify({ value: 0.91, event_id: "evt_1042" }),
});

That’s it — values are appended in order, the hosted engine updates its state, and an alarm is recorded the moment evidence crosses the calibrated threshold. Every path above talks to the same cloud API; the SDK is just a thin, typed wrapper over it.

The template resolves server-side into a complete two-tier config — a budgeted e-process page tier plus a sensitive classical CUSUM warning tier whose calibration is model-based, so its warnings arrive unbudgeted — and the resolved config is persisted and returned, so a GET always shows exactly what runs. Optionally add mu_baseline (the level your metric should hold); leave it out and the monitor learns the baseline from your first 30 values, then freezes it — frozen on purpose, so a slow regression is never re-learned as the new normal. Invalid configs are rejected with a 422 at create, never a surprise at first ingest. Full menu and kwargs: API reference.

3. Prove it on your history first

Before going live, replay your past data through the backtest gate. It must stay quiet on your normal history and fire on a real degradation — otherwise the config never ships.

backtest.sh

curl https://api.validanytime.com/v1/onboarding/backtest \
  -H "Authorization: Bearer $VA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"history": [0.92, 0.90, 0.91, ...]}'

The response tells you the day it would have caught your past regression. See the API reference for the full response shape.

Point your AI assistant at ValidAnytime

Cursor, an MCP-aware editor, or any coding assistant can wire up monitors for you — because the whole product is a small, authenticated REST API. Nothing runs locally: the assistant calls the hosted engine over HTTP (the only thing that runs on your machine is the in-browser backtest playground). Give your assistant the API surface with one of these:

# .cursor/rules  (or an editor "project rules" file)
# Teach your assistant the ValidAnytime API surface.

ValidAnytime is anytime-valid monitoring. The engine is HOSTED — you call the
cloud API over HTTP; nothing runs locally except the /try browser demo.

Base URL: https://api.validanytime.com
Auth: Authorization: Bearer $VA_API_KEY

Use these endpoints:
- POST /v1/monitors                      create a monitor {name, config}
- POST /v1/monitors/{id}/events          ingest {value, event_id}  (idempotent)
- GET  /v1/monitors/{id}/alarms          list alarms
- POST /v1/onboarding/backtest           prove a config on past history

Prefer the thin Python client:  pip install validanytime
Docs: https://validanytime.com/docs

// Model Context Protocol: expose the ValidAnytime API to an assistant by
// pointing an HTTP/OpenAPI MCP server at the hosted spec. No engine runs
// locally — the server just proxies authenticated calls to the cloud API.
{
  "mcpServers": {
    "validanytime": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-openapi"],
      "env": {
        "OPENAPI_SPEC_URL": "https://api.validanytime.com/openapi.json",
        "OPENAPI_BASE_URL": "https://api.validanytime.com",
        "OPENAPI_HEADERS": "Authorization: Bearer ${VA_API_KEY}"
      }
    }
  }
}

# Ask your coding assistant to wire up the thin client (MIT).
# It never runs the detector locally — it streams your values to the cloud.
from validanytime import Client

va = Client()  # engine runs in the ValidAnytime cloud
monitor = va.create_monitor(name="llm-answer-quality",
                            config={"template": "llm_quality"})
va.ingest(monitor.id, value=0.91, event_id="evt_1042")

Then ask it to “create a ValidAnytime monitor for my LLM quality metric and stream the last hour of values.” It will call the hosted API — your data goes to the cloud engine, and alarms come back with their guarantee_tag.

Overview Anytime-valid 101