Verdict captures supported provider calls, stores traces locally by default, and helps you detect calibrated PASS/FAIL quality drift by workload. Fisher's exact for binary dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect sizing, and Benjamini-Hochberg correction.
verdict.init() wraps supported Anthropic, OpenAI, and Google SDK calls. No proxy in front of your traffic. Metadata capture is immediate; content capture is opt-in and redacted before storage.
Manual @verdict.trace() spans are available for retrieval, reranking, and tool calls. Full agent-run stitching and task-success metrics are v1 roadmap items.
import verdict from anthropic import Anthropic verdict.init( service_name="checkout-agent", storage="sqlite:///./verdict.db", ) # Use the SDK normally. Supported calls are captured. # Run the judge and drift workflows with your own # provider key when you want quality scoring. client = Anthropic() resp = client.messages.create( model="claude-haiku-4-5", max_tokens=512, messages=[{"role": "user", "content": q}], )
Verdict separates capture from evaluation. You can start key-free with metadata, structural checks, and semantic drift; add a provider key when you want judge-based PASS/FAIL scoring and workload-specific calibration.
Record provider, model, tokens, latency, cost estimate, finish reason, errors, and optional redacted prompt/response content from supported SDK calls.
Inspect refusal rate, JSON validity, response length, latency, token usage, and semantic drift with the built-in lightweight embedder.
Sample traces, label PASS/FAIL dimensions yourself, then measure judge agreement before trusting drift alerts for that workload.
Cluster similar prompts so quality shifts are evaluated by workload instead of hidden inside one global average.
Compare recent windows against a reference baseline and emit signals only when statistical and practical effect-size gates both clear.
The repo includes live capture checks, a synthetic regression injector, pairwise judge-alignment harnesses, and a rubric-alignment workflow for measuring agreement against your own labels.
The defensible claim is deliberately narrow: calibrated PASS/FAIL drift monitoring on real LLM traces, with per-workload judge calibration. Public benchmarks are sanity checks, not proof that a judge is calibrated for your workload.
| Workflow | What it checks | Key required? |
|---|---|---|
| Smoke test | storage, schemas, basic pipeline | No |
| Live capture check | real SDK capture paths | Provider key |
| Rubric alignment | judge agreement vs your labels | Provider key |
| Regression injector | known corruptions through the eval path | Optional |
| Dashboard | local trace and drift inspection | No |
Judge quality depends on the model, rubric, and workload. Run the alignment scripts on your own labeled traces before relying on alerts.
Provider, model, or prompt changes can shift refusal behavior. Verdict gives you captured traces and structural signals so a reviewer can inspect the change by workload.
Use pairwise comparison and workload grouping to inspect where a candidate model helps, hurts, or needs more labels before you switch production traffic.
Prompt changes can alter response length and token usage. Verdict keeps quality, cost, latency, and error metadata in the same trace store for faster investigation.
SDK is Apache 2.0. SQLite storage runs locally with zero ops. Postgres and a local dashboard are available when you need shared inspection.