Evidence before release.
A release workbench for a fictional operations analytics assistant.
All records and outputs are invented. Latency is simulated. Fixture pass rates are not model accuracy, business savings, or production validation.
Why do some cases fail?
These are deliberately written sample responses, not live AI answers. The baseline includes known mistakes so you can see which checks catch them. FAIL describes a sample response that misses a check; it does not mean this website is broken.
The PASS and FAIL labels below evaluate the sample answers against the task requirements.
Fixture checks passed
Golden facts matched
Release decision
Production approval
Requires live evaluation, human review, operational controls, and accountable sign-off.
Golden cases
Select a case to inspect its evidence.
Metric definitions
Case pass count uses all golden cases as its denominator, including missing or malformed responses. A case passes only when it has zero failures. Golden-fact matches use all expected fact keys as their denominator; missing and incorrect values receive zero credit. Extra fact keys are critical failures but do not enlarge this denominator. Zero denominators display N/A, never 100%. Failure totals count events, not affected cases. There is no aggregate measured-latency or model-accuracy metric.
Release policy & ownership
Deterministic fixture gate
Every case in the selected golden version must pass, with at least one case. An empty suite is HOLD. No severity overrides are supported. Zero critical failures: schema, incorrect facts, unsupported structured assertions or citations. Zero major failures: incomplete facts/citations or simulated latency above 1,500 ms.
Human release judgment
Product owner defines business utility. Evaluation owner maintains the golden set. Engineering owns runtime and rollback. A reviewer must assess free-text claims: this checker cannot detect arbitrary hallucinations, misleading framing, or omitted context.
Passing fixtures permits review of the next experiment only. Future rollout: offline live-model evaluation → shadow use → limited read-only pilot → monitored expansion, with rollback on critical factual or permission failures.