Agents & MCP

claude mcp add validanytime \
  --env VALIDANYTIME_API_KEY=va_your_key_here \
  -- uv run --directory /path/to/validanytime-cloud/mcp validanytime-mcp

{
  "mcpServers": {
    "validanytime": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/validanytime-cloud/mcp", "validanytime-mcp"],
      "env": {
        "VALIDANYTIME_API_KEY": "va_your_key_here",
        "VALIDANYTIME_API_URL": "https://api.validanytime.com"
      }
    }
  }
}

Set up a ValidAnytime monitor for this metric, and pick the config from the data.

1. Get a sample of the metric's recent values — from the attached data, or ask
   me where to find it.
2. Call validanytime_suggest_config(sample): it reads the shape and returns a
   config already ratified by the backtest gate. (Know the use case already?
   Pass use_case, or call validanytime_recommend_config first to browse the
   matrix without sending data.)
3. Only if the verdict's "passed" is true, create the monitor with that EXACT
   config: validanytime_create_monitor(config=...). If it is false, do not go
   live — report why, then retry with a longer history or a use_case hint.
4. Backfill the history with validanytime_ingest (keep the original order; pass
   event ids so retries stay idempotent), then list alarms and report by tier.

Set up monitoring for my nightly eval scores with ValidAnytime.

1. Create a monitor from the llm_quality template (leave mu_baseline unset —
   it is learned from the first 30 points, then frozen).
2. Backfill the attached history with validanytime_ingest (keep the original
   order; pass event ids so retries stay idempotent).
3. List alarms and report them BY TIER: page-tier alarms are within the
   monitor's stated false-alarm budget; warn-tier alarms ("heuristic_adaptive")
   are unbudgeted early hints — do not treat a warning as a page.

Before this monitor goes live, prove the config on my own data.

Run validanytime_backtest on the attached metric history. Read the verdict
fields, not the prose: "passed" must be true, "false_alarm_on_normal" must be
false. "warned_on_normal" does not fail the gate — the warning tier is
unbudgeted by design. If the gate fails, say which monitor fired inside the
normal stretch and at which point, and retry with the conservative template.

An alarm fired. Use validanytime_explain_alarm and answer three things:

1. Which tier — a budgeted page or an unbudgeted warning?
2. What is the guarantee: quote the guarantee_tag and, for page-tier alarms,
   the e-value and what it bounds.
3. What would you do next — and if it is a warn-tier alarm, say explicitly
   that it is a hint to investigate, not a page within a budget.

name: metric-regression-gate
on: [pull_request]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Backtest the eval history
        id: gate
        uses: validanytime/validanytime-gate@v1   # publishes at launch
        with:
          csv: evals/nightly_judge_scores.csv
          column: judge_score
      - run: echo "${{ steps.gate.outputs.summary }}"

pip install validanytime validanytime-detectors   # publishes at launch

validanytime init                    # starter script, no network
validanytime backtest metrics.csv    # offline gate verdict, exit 1 on fail