Cognifity builds observability tools for AI in production. Verdict — our first product — works with existing Verdict-compatible traces or captures supported LLM calls, then helps teams measure quality drift with calibrated PASS/FAIL rubrics.
Most LLM observability starts with latency, tokens, and errors. Verdict adds quality monitoring: start with existing compatible traces or capture supported provider calls, group similar prompts, evaluate sampled responses with a rubric, and compare recent behavior against a reference window.
Start with an existing Verdict-compatible SQLite or PostgreSQL store. If you need capture, verdict.init() wraps supported Anthropic, OpenAI, and Google SDK calls without putting a proxy in front of your traffic.
Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ for effect-size gating, and Benjamini-Hochberg correction across dimensions.
Calibration scripts compare judge decisions against your own human labels. Public benchmarks are treated as sanity checks, not proof that a judge is calibrated for your workload.
Production AI teams already have logs, latency charts, and token dashboards. Cognifity products focus on the harder operational question: did behavior change for the workloads your users actually run?
Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect-size gating, and Benjamini-Hochberg correction across dimensions.
The repo includes workflows for judge calibration, live capture checks, and injected drift tests. We say what v0 catches today and keep agent-run outcomes in the roadmap until they ship.
Verdict is Apache 2.0. Run it locally, inspect the code, keep your traces in your own storage, and bring your own provider key for judge-based scoring.
Storage adapters for SQLite/Postgres/in-memory and provider adapters for Anthropic, OpenAI, and Google keep evaluation logic separate from provider SDKs.
Verdict captures the calls, tracks structural signals, and helps you compare recent windows against a baseline so a reviewer can inspect affected traces quickly.
Use pairwise comparison and workload grouping to inspect where a candidate model helps, hurts, or needs more labels before you switch production traffic.
Captured traces include tokens, latency, estimated cost, finish reason, errors, and optional best-effort-sanitized content, so cost and quality investigations start from the same evidence.
Watch the 72-second walkthrough, then install the latest compatible public-alpha packages directly from PyPI. If you need capture, add the maintained agent skill after the packages. No source checkout is required.