Your model changed and nobody told you
A term-sheet extractor has run in production for months. Nobody touched the code. On Monday it starts booking the wrong barrier levels.
Nothing in the application changed. The model behind it did. When a model is called through a hosted API, the thing answering today may not be the thing that answered when you tested it. In What MAS's AI guidelines ask of a markets business, I described vendor updates as a change-management event. This post is about how to notice one.
It has been measured
In 2023, Lingjiao Chen, Matei Zaharia and James Zou ran the same tasks against the March and June 2023 versions of GPT-4 and GPT-3.5 and published the differences in How Is ChatGPT's Behavior Changing over Time? Two of their findings are useful here.
Capability moved on a fixed task. On 1,000 questions asking whether a number is prime, GPT-4's accuracy fell from 84.0% in March to 51.1% in June. GPT-3.5 moved the other way, from 49.6% to 76.2%. Same prompts, same questions, same model name.
Format moved, and that broke the pipeline. On 50 code-generation problems, the share of GPT-4 outputs that ran directly fell from 52.0% to 10.0%. The authors traced this to the June versions adding extra non-code text: “the extra triple quotes render the code not executable.” The output no longer fitted what the next step expected.
The second finding is the one production teams tend to meet first. A model can get better at the task and still break your system, because your system depends on more than the answer being right.
Same score, different model
Here is a synthetic regression set for a term-sheet extractor: 20 term sheets, four fields each (underlying, coupon, barrier, maturity), so 80 field checks. We run it against the old model version and a new one.
| Measure | Old version | New version |
|---|---|---|
| Fields correct | 74 of 80 (92.5%) | 74 of 80 (92.5%) |
| Term sheets fully correct | 14 of 20 | 14 of 20 |
| Underlying fields wrong | 1 | 0 |
| Coupon fields wrong | 3 | 1 |
| Maturity fields wrong | 2 | 1 |
| Barrier fields wrong | 0 | 4 |
| Outputs a strict JSON parser accepts | 20 of 20 | 0 of 20 |
On the headline score, nothing changed. Underneath it, eight field outcomes changed: four errors were fixed (one underlying, two coupons, one maturity) and four new ones appeared. All four new errors are barrier levels, the field that decides whether a client's capital is at risk.
And in this example, every new output arrives wrapped in a Markdown code fence. The JSON inside is valid, but a strict parser rejects all 20. Depending on how the pipeline handles a parse failure, that is either an outage or, worse, a silent fallback to empty fields.
A single accuracy number hides both problems. It averages away which cases moved, and it is usually computed after cleanup that production may not do.
Compare the two versions
Illustrative outcomes. The same 20 synthetic term sheets, scored for each version. Outcomes are fixed for the example; nothing here calls a model.
74 of 80 fields correct (92.5%)
8 cells will change in the new version (outlined).
Raw output, first term sheet
What to notice: The accuracy line does not move when you switch versions. The outlined cells and the strict-parser result do. Those are what a release check should report.
Four controls
Pin the version you tested. If your provider offers a fixed version identifier alongside a moving alias, call the fixed one in production. Treat a move to a newer version as a release: planned, tested and approved. Pinning delays change rather than preventing it, because old versions are eventually retired, so the next three controls still matter.
Keep a regression set that looks like your work. A few dozen real-shaped cases with known answers, scored field by field. Include the cases that matter most, such as barriers, not just the easy ones. This is the domain evaluation set the MAS guidelines point towards when they expect firms to test third-party AI on their own use cases.
Compare case by case, not score to score. Report what changed: which cases were fixed, which broke and which fields they touch. “Accuracy unchanged” should never be the whole release note. In the example above, the useful line is “four new barrier errors”.
Test the contract, not just the content. Check that raw outputs parse exactly as the next step expects, before any cleanup. Format checks are cheap and deterministic, and they catch the kind of failure behind the code-generation drop in the 2023 study.
When a provider says a model was improved, the useful question is: improved at what, and what did it stop doing? Only your own regression set can answer that.
The regression example is synthetic. Figures for GPT-4 and GPT-3.5 are as published by the cited authors.
Further reading
- Lingjiao Chen, Matei Zaharia and James Zou, How Is ChatGPT's Behavior Changing over Time?, arXiv 2307.09009
- Evaluating AI assistants beyond an accuracy score
- What MAS's AI guidelines ask of a markets business