Verdict / Evidence
Benchmarks and evidence
Every claim in these docs maps to a repository artifact with a status: verified locally, observed on a dated live run, or not claimed. Where the evidence is a fixture or an estimate, the page says so.
How claims are governed
Status: ShippedThe repository keeps a claims ledger and a proof matrix. Each claim records its status, evidence paths, observation date, confidence, known objections, and the exact wording allowed in public. A validator script checks both files, and the most recent audit (2026-09-06) found no missing evidence paths.
The authority order is deliberate: current source and tests first, then CI and runtime evidence, then versioned documentation. A checked task, an issue checkbox, an old session summary, or a workflow definition is not evidence on its own.
Verified locally
Status: Shipped| Claim | How to check it |
|---|---|
| Hard eligibility runs before advisory ranking; excluded candidates cannot return. | tests/test_eligibility_gate.py tries to reintroduce an excluded candidate through the ranker. |
| Missing or stale runtime truth cannot silently authorize protected work. | tests/test_availability_cache.py feeds stale, contradictory, malformed, and missing observations. |
| A credential-free quickstart makes a deterministic decision with explicit exclusions. | verdict quickstart --non-interactive --dry-run |
| Local benchmark fixtures and evidence bundles are reproducible and digest-verifiable. | tests/test_benchmarking.py and tests/test_evidence_bundle.py |
| Failover and replay work without a network. | verdict failover-proof --json, then verdict replay |
Cost comparison: a deterministic mock
Status: Shippedpython -m verdict.routing_demo --mock routes exactly 100 fixed requests and compares a class-aware route with a baseline that uses the costliest qualified model, based on fixed Opus, Sonnet, and Haiku price estimates. The recorded fixture reports roughly $0.16 routed against a $0.52 baseline.
Dated live observations
Status: Experimental- Context lift. On 2026-08-31, one free-tier model failed an exact check unaided and passed it with a compiled context pack. That is one paired observation, described in Context Intelligence.
- Catalog snapshots. Gateway catalog records from July 2026 are bounded historical observations. By their own limitations, catalog membership is not liveness, authorization, quota, or eligibility.
- Paired savings bench. The bench can claim a saving only from executed, receipt-bound runs: both arms executed on the same input hash, observed cost, provider-bound identity, quality checked on both outputs, no cache hit. Its default mode is a labeled simulation that cannot claim anything. These docs publish no live savings figure.
What is not claimed
The evidence ledger explicitly does not approve the following without a reproducible artifact that defines the metric, baseline, environment, date, and raw result:
- Production readiness, adoption, or usage figures.
- Latency, throughput, or percentage-improvement figures.
- Model-quality or performance leadership.
- Live-provider quality, availability, or cost derived from fixture runs.
- That every release gate has passed. CI workflows are defined for tests, lint, types, security, install, build, and CodeQL, but a workflow definition is not a passing run.
Sources
Primary material in the public Verdict repository.