Evaluating AI assistants beyond an accuracy score

It is easy to trust an answer that reads well. The writing is clear, the numbers look sensible, and there are a few links at the bottom. We skim it, find what we were looking for, and move on. That is often how an AI assistant earns our confidence, even when parts of its answer have never been checked.

The trouble is that an answer can be right in one place and wrong in another. It might get the total right but invent the reason behind it. It might answer most of the question and quietly skip the difficult part. An accuracy score can help us compare results, but it tells us less than we think unless we know exactly what was counted as correct.

Take a simple, hypothetical example. An operations analyst asks, "How many orders are still open, what are they worth, and why are they waiting?" There are two open orders worth 120 and 230, and a closed order worth 80. The records show their status and value. They say nothing about staffing or delays.

The assistant answers, "Two orders are open, worth 350 in total. They are waiting because the team is short-staffed."

The arithmetic is fine. The explanation is made up. If we check only the count and total, this answer passes. But the analyst could use that explanation to make a staffing decision. A better answer would give the count and total, then say that the records do not explain the delay. It could also suggest what information would help answer that part of the question.

This is why I would start evaluation with the task, rather than the score. What does the user need to know or do? Which parts can we check against records? Where should the assistant admit that it does not have enough information? Writing down those expectations helps us decide what a good answer looks like before we start grading one.

For numbers and other clear facts, use code where it makes sense. Add up the orders again. Check the date range. Confirm that the assistant has not included a cancelled item or left out a refund. These checks are quick to repeat, and when they fail, we can usually see why.

Then look at the rest of the answer. Did it cover everything the user asked? Did it add claims that the records cannot support? Do its links point to useful evidence, and does that evidence actually back the sentence? A real citation can still be attached to a claim that does not follow from it. The presence of a link is a useful first check, not the end of the job.

It helps to keep these results separate. Knowing that the numbers are correct is useful. Knowing that the answer is complete and supported is useful too. Rolling everything into one score can hide the very mistake we need to notice. We should be able to open a failed answer and understand what went wrong without having to decode the dashboard.

Exact checks have limits of their own. A test can reject a perfectly good answer because it expects a particular phrase or too many decimal places. It can also reward the wrong answer if the expected value was written using the wrong business rule. The checks need review just as the assistant does. If two people who know the work disagree about what should pass, the task or the test may need to be clearer.

For the parts that need judgment, people still matter. Someone who understands the work can spot a misleading explanation, a missing qualification or an answer that is technically correct but unhelpful. Another AI model can help review more answers, especially if we give it clear questions to answer. But before trusting its grades, compare them with a set of answers that people have reviewed. Look closely at what it wrongly accepts and rejects. Giving the judge a way to say "I can't tell from this evidence" is better than asking it to sound certain about everything.

The answer is also only part of the experience. Did the assistant finish the task? Did it use the right records? Did it get stuck, or make the user repeat information unnecessarily? When tools are involved, check what actually happened. An assistant saying it updated a record is not proof that the record changed correctly.

Finally, decide which failures matter enough to stop a release. An awkward sentence in an internal draft and a wrong recommendation sent to a customer do not carry the same risk. Some problems call for a fix before anyone uses the assistant. Others may be acceptable in a limited rollout with review and monitoring. That choice should be clear to the people responsible for the product.

A good evaluation gives us reasons to trust an assistant for particular work, and reasons to be careful elsewhere. As the prompts, tools or data change, keep checking those reasons. The most useful result is not a reassuring number on its own. It is a clearer understanding of what the assistant gets right, what it misses and how those mistakes affect the person using it.

Further reading

NIST: Generative Artificial Intelligence Profile

Anthropic: Demystifying evals for AI agents

For a small illustration, explore the synthetic evaluation demonstration.