Why great AI scores can fail in production
Imagine a delivery company builds an AI model to warn its team about parcels that may arrive late. In testing, the model seems remarkably accurate. Once the team starts using it before parcels leave the depot, it performs much worse. One column in the test data explains the difference: whether the customer was eventually given a refund.
A refund can be closely associated with a late delivery. That does not make it useful for predicting the delay in advance. The refund is usually recorded after the problem has happened, while the team needs its warning before the journey is complete. Giving the model that column lets it use later information to answer an earlier question. This is one form of data leakage: information enters an evaluation that would not be available when the prediction must actually be made.
Start by writing down when the decision happens. If the team makes its prediction when a parcel leaves the depot, it can use the planned route and the weather forecast available then. It cannot use a later complaint, refund or final delivery status. An input that reveals the answer, or something recorded because of the answer, is often called target leakage. The target is simply the thing we are trying to predict—in this case, whether the delivery will be late.
The interactive example below makes that timing problem visible. It also shows why the choice of records used for checking predictions matters. You do not need to build a model to explore it: the rule, records and calculations are all shown.
Watch future information inflate a score
Imagine you run a delivery service and want to predict which parcels will arrive late. We will use eight fictional deliveries to show how information that becomes available only after a journey can make a prediction look better than it really is. The example uses a rule that we wrote ourselves. It is not an AI model that has learned from delivery records.
You can choose which pieces of information the rule uses. With the two starting choices selected, it predicts a delay when the route is long or a storm is forecast. We check those predictions against what happened in four of the deliveries. Two predictions are correct, so the result is 2 out of 4, or 50%. Now select the refund option and watch the result change. When that option is selected, this deliberately flawed rule uses the refund record instead of the route and weather information.
Loading the interactive example.
Choose what the rule is allowed to know
Prediction time: route + forecast available → journey → actual delay + refund recorded later
The result rises to 4 out of 4, or 100%. In these fictional records, every late delivery received a refund and every on-time delivery did not. The refund therefore gives the rule the answer. But the delivery team could not have known about that later refund when deciding whether the parcel would arrive late. The score improved because we gave the rule a clue from the future, not because it became better at predicting an unfinished journey.
Inspect the tested predictions
| Day | Prediction | Actual delay | Correct? |
|---|
The eight fictional deliveries
Each row is one fictional delivery. Day 1 happened before day 2, and so on. The starting option checks predictions on days 5 to 8, for customers C and D. Days 1 to 4, for customers A and B, stand in for the earlier records a real model might learn from. Our rule is already written and does not learn from those rows; the two groups simply let us illustrate how an evaluation can be arranged.
| Day | Customer | Long route | Storm | Later refund | Delayed |
|---|
See the rule and which deliveries are checked
The rule does not change as it sees records. If the refund option is off, it predicts a delay when at least one selected route or weather value is 1. If the refund option is on, it uses the refund value as the whole prediction. The result is the number of correct predictions divided by the four checked deliveries. The later-day option checks days 5 to 8; the alternating-day option checks days 2, 4, 6 and 8.
Remove the refund option, then choose the alternating-row setting. That setting checks days 2, 4, 6 and 8 instead. The score changes because different rows are tested. The fixed rule does not learn or benefit from customer familiarity. The same customers now appear in both groups. This may be a reasonable question if you expect to serve familiar customers, but it cannot tell you how well a model will perform for customers it has never encountered. Eight fictional records are useful for seeing the distinction, not for estimating the performance of a delivery service.
The controls run locally. Existing site analytics remain enabled; inputs and derived results are masked for Clarity, and no custom events send these controls.
A real model can encounter leakage even when every input was recorded early enough. Suppose you want to use delivery records to learn a rule, then check it on other deliveries. Those checking records need to remain separate while you build and choose the model. Otherwise, information about them can influence the model before it takes its test. This is usually called train/test leakage: training is the learning stage, and the test is the later check on records that were not used to make those choices.
The separation applies to preparation as well as to the model itself. You might fill gaps in a column using its average, convert customer categories into numerical values, or choose the most useful columns. These steps are often grouped under the name preprocessing. If you calculate those choices using the whole dataset, you have already let the checking records influence the process. Work out the preparation steps using the learning records, then apply those same choices to the checking records without recalculating them.
The same care is needed when you try several versions of a model. Choosing and adjusting its settings is often called tuning. A common method, cross-validation, takes turns reserving different portions of the records for checking. Each reserved portion is sometimes called a fold. During each turn, the model and its preparation steps should use only the other records. Keep another final set aside until you have chosen the approach, so repeated tuning does not gradually turn that final test into another part of model building.
Choose the separation to match the job. To estimate performance next month, learn from earlier dates and check later ones; this is a chronological split. To estimate performance for new customers, keep all records for a customer on one side of the split; this is a grouped split. A random selection of rows answers a different question, particularly when the same customer or almost-identical order appears many times. There is no universally correct split without a clear description of how the model will be used.
For generative AI, another problem arises when the test questions were already present in the material used to train the model. The model may have seen a question, its answer or a close version of both while learning. A high score can then reflect familiarity with the test rather than an ability to handle unfamiliar work. This is called benchmark contamination. A benchmark is a shared collection of tasks used to compare systems. Check for duplicates when the data is available and create separate task examples for your own workflow. Changing a few words in a public question does not guarantee that it becomes an independent test.
Privacy is a separate concern. Some models can reproduce identifying or sensitive information from their training material. This is often described as memorization. Published extraction research shows that such material can sometimes be recovered under the conditions studied. It does not show that every model will reveal every record. Any test of disclosure needs an authorized scope, safe test data and a clear plan for handling unexpected sensitive output.
These problems need different remedies. A better train/test split does not remove personal information from the training collection. Removing personal information does not prove that a benchmark is independent. Limit unnecessary sensitive data, control access to training collections, review repeated records and assess disclosure risk where appropriate. Restrictions on what a model may output can reduce exposure, but they are not proof that the model has forgotten the underlying information.
When a score looks unusually good, ask someone to reconstruct one prediction using only the records that existed at the time of the decision. Then check how the evaluation records were kept separate while the model was prepared and chosen. If correcting leakage lowers the score, keep the earlier result and explain what changed. A less impressive result that reflects the real task is more useful than a high score built on information the system will never have when it needs it.
Further reading
scikit-learn: Common pitfalls and data leakage
Research paper: NLP evaluation and benchmark contamination
Research paper: Extracting training data from large language models