How do you test AI applications?

Can AI applications be tested? No, at least not by asking, ‘Is this answer correct?’. After all, nobody can answer that question when it comes to a chatbot or a translation system. There isn’t just one correct output; rather, there are many that would be acceptable, and many that wouldn’t. Nevertheless, such software can still be validated. You simply have to stop looking for the one correct answer and start narrowing down the range of acceptable outcomes.

Our colleague Christoph Menke has described how this works in a paper. In conversation with CTO Alexander Hofmann, he explains why no single test can prove correctness, why it is dangerous for one AI to act as a referee over another, and why the first step is much smaller than feared.

Christoph, what is your paper about?
In short: systems with AI components behave unpredictably. The same input can produce two different outputs, and what is correct depends on the context and is often not even known. This undermines the fundamental assumption of traditional quality assurance. Such software can still be validated, but only if we stop trying to verify a single ‘correct’ output.

Let’s stick with this basic assumption. Which one, exactly, is being undermined?
Comparing an output against a single expected value. We testers call this value the target. A test has always worked like this: I know the target in advance, I send in an input, I compare it and decide whether it has passed or failed. A sorting function always produces the same, unambiguously correct order; that can be checked. With a chatbot or a translation system, however, there is no single authority to decide whether a particular output is correct.

If no one can say what is correct anymore: is everything then permitted? 
No, and that is the point where most people go astray. The target does not disappear; it changes form. A single value becomes a set of permissible answers. I work this through in the paper using an example: a travel planning chatbot for end customers that recommends accommodation from its own catalogue. If someone asks for a three-day break by the sea, there are many recommendations that would be acceptable, and many that would not. This set exists; it’s just that no one has written it down in full. Instead of verifying the one ‘correct’ answer, we narrow down this set.

Narrowing down rather than verifying. How does one do this in practice? 
With six types of checks, each providing a different kind of evidence:

For the bottom three, the target can be set in advance, so they run automatically. Fixed rules, for example – that is, statements that must apply to every response. In the case of the travel chatbot, these are phrases such as: ‘The chatbot deals exclusively with travel topics, and recommendations are made solely from our own catalogue.’ Added to this are relationships between responses: if I rephrase the same enquiry, the result must be similar. I can check this without knowing the correct answer. 

With the top three, this isn’t possible, because nobody knows what attacks will come tomorrow, where operations will shift, or whether a recommendation will actually help. This requires runtime protection mechanisms, monitoring during operation and, ultimately, a human to assess whether the product benefits the customer.

Six types of testing sounds like a lot of effort for a result that still isn’t proof.
No single one proves correctness; that’s true. Together, they capture the behaviour. And behind that lies the most uncomfortable insight from the paper: the harder the proof, the less it says about the benefit. The fixed rules prove something but say nothing about the quality of a recommendation. The human being says everything about the quality and proves nothing. At the bottom, one proves that the rules hold true. At the top, one finds out whether it benefits the customer. 

Many solve precisely this evaluation problem by having one AI grade the other.
And that is precisely where it becomes dangerous. Human evaluation does not scale, which is why this is common practice. But such an evaluator is vulnerable: you can slip an instruction into the text being assessed, and it will grade your own provider more leniently. The obvious reaction – having a second model cross-check it – doesn’t help either. AI models make correlated errors; in other words, they get things wrong in similar places, and the better they become, the more this happens. A second model feels like an independent opinion, but it isn’t. 

What, then, is the alternative? 
An anchor that does not originate from a model. That is why my checklist includes the question of whether at least one link in every verification chain is tied to a fact, a rule or a database. If not, the system is verifying itself. It is perfectly possible to use AI for verification; one simply must not let AI make the decisions.

So far, we’ve been talking about systems that provide answers. What changes when they take action?
The cost. Let’s stick with the travel chatbot: up to now, it recommends a hotel. An agent books it and sends out the confirmation. I correct an incorrect recommendation in the text. I cancel an incorrect booking, pay the fees and explain the situation to the customer. All six types of verification take place after the output has been generated, and by then it’s too late. That’s why there’s a type of verification added at an earlier stage, which asks: What is the system actually allowed to do? In other words, an inventory of all actions with external consequences, the question of reversibility for each action, and authorisation before every step that cannot be undone.

And where, in practice, do such agents go wrong?
Not where most people look. An analysis of over 1,600 processes shows that the largest group, at 41.8 per cent, relates to the target specification; in other words, the agent tackles the wrong task or loses sight of the target amidst the many steps. The second-largest group, at 36.9 per cent, arises during the handover from one agent to the next – precisely where, in most projects, nobody looks. 

Your paper states that, ultimately, the real bottleneck is not a testing technique after all.
The market distinguishes between two things. Testing is a technical matter; verification is a compliance matter. With this type of software, the two coincide because there is no proof of correctness. The collected evidence constitutes the assurance. That is why I propose the release document: it begins with the question of which types of errors are tolerable and to what extent; then each type of test provides its evidence, and the evidence is cross-checked against this. This transforms the intractable question of ‘enough’ into a manageable one: is the chain of evidence complete and traceable?

However, this document does not answer itself. No tool can decide which type of error is tolerable, nor can it determine whether a piece of evidence is genuinely valid or merely appears to be complete. The real bottleneck is the human factor – more specifically, the ability to question a plausible result. Code is being produced faster than ever before, and the toolkit for testing is growing alongside it, but both are of little use without people capable of making judgements using them. For anyone coming from a testing background, this is good news: The role is becoming more demanding, not redundant, and anyone who is now seriously engaging with how to secure AI applications has a few very exciting years ahead of them.

And yet, in the end, a residual risk remains.
Of course. But that is not unique to AI. Complete testing is not possible, and testing demonstrates the presence of faults, never their absence. This has been a fundamental principle of software testing for decades, and nobody builds less software because of it. We have never proven that software is bug-free. We have demonstrated that its residual risk is acceptable. What is changing with AI is not the question of whether it can be secured, but the toolbox.

Where does a team start when it realises that its testing approach is no longer viable?
It’s simpler than it sounds. You note down where in the system a model actually makes decisions and where it does not. Because not everything is AI. Even in the case of a travel chatbot, catalogue searches, the booking process and payment integration behave deterministically, have a target value in the traditional sense, and continue to be tested in the traditional way. This is an architectural question and takes a morning. Everything else follows from this.