How to read the evidence

This prototype uses fictional business operations records and written sample responses. The deterministic checks actually run in Python; model behavior and response latency are simulated.

Architecture

Versioned synthetic records and golden expectations feed the evaluator alongside recorded outputs. Python emits a JSON report. A static browser interface displays the evidence and decision. It cannot call a model or approve production release.

Two different counts

Fixture checks passed means cases with zero failures divided by all declared golden cases. Missing and malformed responses remain in that denominator. Golden facts matched means scalar keys present with the exact expected type and value divided by all expected keys. Missing or incorrect facts earn zero credit. Extra fact keys create critical failures but do not enlarge the expected-fact denominator.

The recorded regression demonstrates why both views matter: 14 of 14 expected facts match, yet one extra unsupported claim blocks the gate. Failure totals count events, not affected cases. Zero denominators show N/A, and an empty suite is HOLD.

Release rule

At least one case, every case passing, zero critical events, and zero major events are required. Schema failures, incorrect facts, and unsupported structured assertions or citations are critical. Missing facts/citations and simulated latency-budget misses are major. No severity overrides exist. Each latency budget is 1,500 simulated milliseconds; the exact boundary passes.

Passing the fixture gate allows consideration of the next evaluation stage. Production approval remains false. These results do not measure live-model accuracy, actual model latency, business savings, or production readiness.

Limits and ownership

The checker does not validate the meaning of summary prose. Citation membership does not prove that a narrative is supported. Ten curated cases cannot establish broad reliability. Golden-file schema and unexpected output case IDs are not fully validated; adding cases requires review.

Product defines utility and workflow objectives. Evaluation owns golden coverage. Engineering owns runtime, monitoring and recovery. A release owner accepts residual risk. These roles and rollout controls are proposed, not staffed or implemented.

Extensions

A future live adapter must separate measured provenance from recorded fixtures. The separate permissions demonstration uses mock tools and an application-owned policy boundary. Model-selection work is separate and is not demonstrated here. Any comparison would need repeated comparable measurements, actual costs and uncertainty.