Testing an AI agent: where to start?
An AI agent never answers exactly the same way twice. Here is how to build a first test strategy that holds up: scenario sets, evaluation criteria, guardrails and regression testing.
Testing an AI agent means giving up the comfort of expect(result).toBe(42). The same question can produce two different answers that are both right, or one right and one wrong. That doesn’t make the QA job disappear: it changes its tools and its criteria.
Here is the approach I follow when I start testing an agent or an AI assistant.
1. Write down what the agent must do, and what it must never do
Before the first test, you need a contract. For an agent, it fits in two lists:
- The missions: the tasks it must succeed at (answering a product question, creating a ticket, summarising a document…).
- The red lines: what it must never do (make up a price, disclose personal data, run an action without confirmation, step outside its scope).
The red lines are often forgotten, yet they are what costs the most in production.
2. Build a scenario set, not a data set
A good test set for an agent looks like a list of realistic conversations, grouped into families:
- The happy path: a clear, well-phrased request.
- Rephrasings: the same request worded differently, with typos, in another language, abbreviated.
- Edge cases: an ambiguous request, missing information, an off-topic question.
- Hijacking attempts: “ignore your instructions”, asking for another user’s data, an injection hidden in a supplied document.
Start small: thirty to fifty well-chosen scenarios are worth more than a thousand generated without thinking.
3. Define measurable evaluation criteria
Since you can’t compare word for word, you evaluate properties of the answer:
| Criterion | Question asked | How to check it |
|---|---|---|
| Accuracy | Are the facts right? | Comparison with a reference answer |
| Relevance | Does it answer the question asked? | Human review or an LLM judge |
| Scope | Does it stay in its role? | Rules and forbidden keywords |
| Format | Are the JSON, length and language correct? | Classic assertions |
| Action | Was the right tool called with the right parameters? | Agent traces |
The last two rows are good news: part of testing an agent stays fully deterministic. Check everything that can be checked with classic assertions first.
4. Test the actions, not just the text
An agent doesn’t only talk: it calls tools and APIs, it writes to databases. This is where the tests look most like our usual integration tests. For each scenario, check:
- which tool was called, and in what order;
- with which parameters;
- what happens when the tool fails (timeout, 500 error, empty response).
If your agent exposes its traces (and it should), these checks can be automated like any API test.
5. Run each scenario several times
A scenario that passes once is not a scenario that passes. I run each case several times and look at the pass rate, not just green or red. A case that succeeds 7 times out of 10 is a real problem, even if it was green on the first run.
6. Plug regression testing into CI
Every change to the prompt, the model or a tool can break everything. The scenario set therefore becomes a regression suite:
- run on every change to the prompt or the configuration;
- with an acceptance threshold per scenario family;
- and a report showing what changed compared with the previous version.
Going further
The ISTQB AI Testing syllabus, which I’m currently preparing for, gives all this a solid framework: bias, explainability, robustness testing. I’ll come back to it in future articles, as well as to evaluation by an LLM, which deserves an article of its own.
