Automation KPIs you can actually present at a sprint review

“We have 1,200 automated tests.” The sentence sounds impressive in a meeting, but it answers no useful question. Do these tests protect what matters? Are they reliable? Do they save time?

Since I started setting quarterly automation goals and presenting their progress at sprint reviews, I’ve learned to pick metrics that speak to product people as much as to engineers.

The metrics I track

1. Critical journey coverage

Not code coverage: the share of critical business journeys covered by at least one automated test. If your product has 20 critical journeys and 14 are automated, you’re at 70%. Everyone understands that number, from the PO to the CTO.

2. Stability rate

The share of runs that fail without a real regression (so-called “flaky” failures). It’s the trust metric: beyond a few percent, the team stops looking at the results. I track it per test, to know which ones to fix first.

3. Run time

The time between starting the suite and getting the result. A suite that takes an hour won’t run on every pull request. Parallelisation and sorting tests by priority are often the most cost-effective levers.

4. Bugs found before production

How many defects did the automated tests catch before release during the sprint? This metric shows the real value of automation, and it’s the one that justifies the time invested.

5. Progress towards the quarterly goal

A clear goal (“automate the 10 payment journeys by the end of the quarter”) and a progress curve sprint after sprint. Simple, visual, motivating.

The metrics I avoid

• The raw number of tests: it encourages writing lots of small, not very useful tests.
• The overall pass rate on its own: a 98% pass rate can hide the fact that a single test, the payment one, fails every time.
• Code coverage as a target: useful for developers, misleading as a goal for an end-to-end test strategy.

How to present them

At the sprint review, three minutes are enough:

1. A dashboard with the five metrics and how they evolved over the last sprints (Allure, a spreadsheet or your CI’s reporting tool do the job very well).
2. One highlight: a bug avoided, a test fixed, a journey added.
3. The next goal for the following sprint.

Automation then becomes a product topic, and no longer just a technical task for the QA team.

What’s next?

With AI testing agents arriving, these metrics remain valid, but new ones will be needed: the share of tests generated by AI and validated by a human, the agents’ false positive rate, the time saved on writing tests. I’ll come back to this when the first agents on this site come out of the lab.