What an AI explanation can actually tell you
A delivery model warns that an order may arrive late. Beside the warning, a chart lists route length, weather and backlog, with a bar for each. The chart looks helpful, but what does it actually explain? Does a large backlog bar mean the business can prevent the delay by reducing the backlog, or does it only describe how the model arrived at its prediction?
Those are different questions. An explanation can show which recorded differences influenced a prediction. To decide whether changing a business practice will improve the outcome, we also need evidence about cause and effect. A model may rely on a useful association without knowing what would happen if people acted differently.
It helps to begin with a rule simple enough to inspect completely. The delivery example below lets you change both the delivery and the point of comparison. Watch what stays the same when you change only the comparison. That gives us a concrete way to understand the contribution bars before discussing more advanced methods.
Try explaining one fictional delivery
Imagine a delivery team has a simple rule that gives each delivery a score. A longer route, a storm and more parcels waiting to be handled all raise that score. We will use the rule to explain why one delivery scores higher than another. The numbers are invented score points. A score of 25 does not mean a 25% chance of delay, and the rule was not learned from real delivery data.
The first set of controls describes the delivery we want to explain: a four-hour route, a storm and eight parcels in the backlog. Here, backlog means parcels still waiting to be handled. The second set describes a delivery we choose as the comparison. We call that the reference delivery. The bars show how much each difference between the two deliveries adds to, or subtracts from, the comparison score.
Start by changing Reference route hours from 2 to 4. Leave the selected delivery unchanged. Then compare the result and bars below.
Loading the interactive example.
The selected delivery
The rule starts at 10 points. It adds 2 points for every route hour, 3 points if a storm is present, and half a point for each parcel in the backlog. In symbols: score = 10 + 2 × route hours + 3 × storm indicator + 0.5 × backlog. The storm indicator is just 1 for a storm and 0 for no storm.
The delivery to compare it with
With the starting values, the reference delivery scores 16 and the delivery we are explaining scores 25. Its two extra route hours add 4 points. The storm adds 3, and the four extra backlog parcels add 2. Together those differences explain the nine-point gap: 16 + 4 + 3 + 2 = 25. When you change the reference route from two hours to four, there is no longer a route-time difference. The reference score rises to 20 and the route bar becomes zero. The delivery being explained still scores 25 because you changed the comparison, not that delivery.
The amount assigned to each difference is called its contribution. A negative contribution means that the selected delivery has less of that feature than the comparison, so the feature lowers its score relative to that starting point. It does not show whether the feature is good or bad in every situation, or prove what would happen if the business changed it. These calculations explain the rule we wrote; they do not run the SHAP or LIME libraries.
When two columns carry the same information
Now imagine someone adds a second column called queue length. In this example it always contains exactly the same number as backlog, so it brings no new information. We can write several versions of the rule that divide the original half-point weight between the two columns. Use the slider to choose that division. The two bars receive different amounts of credit, but their combined contribution and the delivery score stay the same.
What this computes, and what it does not
For readers who want the mathematical detail, each contribution here equals the rule’s weight multiplied by the difference between a feature and its reference value. For this additive rule, these are exact Shapley values for a particular fixed-reference comparison: when a feature is left out of a comparison, we temporarily substitute its reference value. That is a definition of the comparison, not a method for filling gaps in a real dataset. It is different from conditional SHAP explanations that account for relationships in a population of records.
LIME creates and checks nearby input examples, then fits a simpler rule to approximate how the original model behaves around that one case. Which examples are considered nearby, and whether they describe plausible deliveries, can affect the explanation. Neither this approximation nor the exact arithmetic above establishes cause and effect in the real delivery service.
The controls run locally. Existing site analytics remain enabled; inputs and derived results are masked for Clarity, and no custom events send these controls.
For a more complicated model, methods such as SHAP provide a way to assign contributions to its inputs. SHAP is based on Shapley values, a mathematical framework for allocating credit among contributors. Rather than taking one feature out in one arbitrary order, that framework considers its contribution across different combinations of included features. The explanation still needs a starting comparison and a rule for what it means to leave a feature out. Different explainers make different choices about those questions.
In practice, the starting point often comes from a collection of comparison records called background data. If you explain a delivery relative to ordinary urban deliveries, you may get different bars from an explanation relative to long rural journeys. The prediction for the delivery can remain unchanged, just as it did in the small example. Record which comparison data and explanation method you used so that someone else can understand the question those bars answer.
Also check the units. Our example adds score points. Another model might explain a probability, while another explains a raw score that must be converted before it becomes a probability. Some classification models use a scale called log odds: it expresses the balance between an outcome happening and not happening on a logarithmic scale. A contribution of 2 on that scale is not a two-percentage-point increase. Ask what number the bars start from, what number they add up to, and what those numbers mean for the person making the decision.
LIME approaches the problem differently. Imagine making many small changes to a delivery record, such as varying route hours or backlog, and asking the original model to predict each changed delivery. LIME then fits a simpler model that approximates those predictions close to the original delivery. That simpler model is called a local surrogate: local because it concerns the neighborhood of one case, and surrogate because it stands in for the more complicated model. It is an approximation, so check whether it reproduces the original model well in that neighborhood.
The changed records need care. If a route category and its geographic coordinates normally go together, changing the category while leaving every coordinate fixed may describe a journey that never occurs. These overlapping relationships are called correlated features. They can affect how SHAP divides credit and how LIME constructs its nearby examples. Try reasonable comparison data and explanation settings, inspect related features together, and check whether the important conclusions remain stable.
An explanation for one delivery is a local explanation. A summary of how a model behaves across many deliveries is often called global. Both can help people inspect a system, but neither replaces checking that its predictions are correct and useful. A clear explanation of a wrong answer is still an explanation of a wrong answer. Show the original inputs beside the bars so a reviewer can also notice an incorrect route or an outdated backlog count.
If the practical question is whether reducing backlog will prevent delays, combine the model review with operational knowledge and an appropriate experiment or causal study. Attribution—the name given to assigning parts of a prediction to its inputs—does not answer that intervention question by itself. A useful explanation helps people ask better questions and challenge a decision. It should not make an uncertain prediction appear more certain than the evidence allows.
Further reading
SHAP: Predictive explanations and causal interpretation
Research paper: A unified approach to interpreting model predictions