Cognifity builds infrastructure for AI in production. Verdict — our first product — captures supported LLM calls, stores traces locally by default, and helps teams measure quality drift with calibrated PASS/FAIL rubrics.
Most LLM observability starts with latency, tokens, and errors. Verdict adds quality monitoring: capture supported provider calls, group similar prompts, evaluate sampled responses with a rubric, and compare recent behavior against a reference window.
verdict.init() captures supported Anthropic, OpenAI, and Google SDK calls. No proxy in front of your traffic. Keep normal SDK code and opt into content capture only when you need it.
Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ for effect-size gating, and Benjamini-Hochberg correction across dimensions.
Calibration scripts compare judge decisions against your own human labels. Public benchmarks are treated as sanity checks, not proof that a judge is calibrated for your workload.
Production AI teams already have logs, latency charts, and token dashboards. Cognifity products focus on the harder operational question: did behavior change for the workloads your users actually run?
Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect-size gating, and Benjamini-Hochberg correction across dimensions.
The repo includes workflows for judge calibration, live capture checks, and injected drift tests. We say what v0 catches today and keep agent-run outcomes in the roadmap until they ship.
Verdict is Apache 2.0. Run it locally, inspect the code, keep your traces in your own storage, and bring your own provider key for judge-based scoring.
Hexagonal architecture, storage adapters for SQLite/Postgres/in-memory, and provider adapters for Anthropic, OpenAI, and Google. Keep the evaluation layer separate from provider SDKs.
Verdict captures the calls, tracks structural signals, and helps you compare recent windows against a baseline so a reviewer can inspect affected traces quickly.
Use pairwise comparison and workload grouping to inspect where a candidate model helps, hurts, or needs more labels before you switch production traffic.
Captured traces include tokens, latency, cost estimates, finish reason, errors, and optional redacted content, so cost and quality investigations start from the same evidence.
Clone the Apache 2.0 repo, run the smoke test, and capture your first supported provider call locally. Enterprise inquiries are reviewed by the team directly.