A flaky test passes and fails on the same code. Same commit, same environment, nothing changed. Run it ten times and you’ll get eight passes and two failures.
That inconsistency makes it worse than an ordinary failure. A real failure tells you something is broken. A flaky failure tells you nothing: you can’t say whether you found a bug or got unlucky. And yet most of us treat the two the same way: we shrug, we rerun, we move on.
Most teams know they have flaky tests. Far fewer know what they cost. Here is a way of finding out.
Why Flaky Tests Survive
Nobody keeps a flaky test on purpose. A build goes red, you recognise the usual suspect, you hit retry, it comes back green, you carry on with your day.
The problem is what those decisions add up to. Atlassian published figures in December 2025: flakiness was behind up to 21% of master build failures in their Jira Frontend repo, around 15% in the backend, and reruns cost more than 150,000 developer hours a year. Microsoft Research put the flaky share of their CI test failures at 13%.
Those are organisations with dedicated build infrastructure teams. If they’re bleeding that much time, your number is probably worse.
The Four Costs
Rerun cost. CI minutes spent on repeat runs, multiplied by your CI rate. It’s the easiest figure and usually the smallest, but calculate it first: a finance team understands it instantly, and it opens the door for the bigger numbers.
Investigation cost. Count the failures someone triaged that turned out not to be bugs. Multiply by average triage time, then by a loaded engineer hour. This normally lands at several times the rerun cost, and it’s the figure that moves a planning meeting.
Escaped defect exposure. Once a suite has cried wolf often enough, engineers rerun instead of investigating, and eventually a real bug rides through as noise. You can’t measure this cleanly, but you can count how many failures last quarter were closed without anyone ever opening them.
Lost trust. No formula for this one. It’s the day your team stops reading the test report. Name it anyway; it’s what makes the other three urgent.
Running The Audit
Two weeks is enough:
- Turn on per-test run history. Most CI platforms log this by default.
- For two weeks, tag every failure: real bug, flaky, or unknown.
- Count reruns and pull requests affected per test.
- Fill in the four costs above.
- Rank by pull requests affected, not by how often a test fails.
In one week of platform data, ten organisations accounted for 82% of every flaky action recorded, and one for 32%. Flakiness concentrates, so your average tells you little about where to start.
A test that fails rarely but blocks your main branch every single time costs far more than one failing constantly in a suite nobody gates on. It’s the same instinct behind risk-based testing, just applied to maintenance rather than coverage.
What You’ll Find
Root causes cluster tightly. Analysing open-source Java projects, Luo et al. attributed roughly 45% of flaky tests to async wait, 20% to concurrency and 12% to test order dependency. The same paper found 54% of async-wait flakes were fixed by awaiting a response instead of a fixed timeout, and 74% of order-dependency flakes by cleaning shared state between runs.
That work underweights one category, because browsers have changed since 2014: selector drift. The test is logically correct, but the element it looks for has been renamed, moved or refactored away. Keep it separate: nothing is racing; it isn’t non-determinism at all. It’s maintenance debt in a reliability costume, and it responds to different treatment.
There’s a second thing that paper can’t see, because it isn’t a property of the test. One parameterised test appeared broadly unreliable until split by data fixture: Belgium failed 56% of runs, Portugal 50%, the Netherlands 40%, France 0%. Three fixtures had incomplete data, and the metric pooled them with the working ones. Before you classify a flaky test, check that it’s one test.
Fix, Delete, or Replace
Fix when the cause is genuine non-determinism in your own code: race conditions, shared state, a teardown that doesn’t mirror its setup. These are bugs in your tests, and they earn engineering time.
Delete when the test covers a path nobody uses, or duplicates coverage you already have. Most suites should be smaller, and deleting is a legitimate outcome of an audit, not an admission of failure.
Replace when the cause is selector drift. Repairing a brittle selector buys you about six weeks before the next one breaks. If a test falls over every time the interface is refactored, the fault lies in how it finds elements rather than in what it asserts, which is the point to look at test execution that adapts when the UI changes, rather than repairing locators by hand.
Quarantine buys time; it doesn’t fix anything, so treat it as a waiting room with a discharge date. And be suspicious of timeout adjustments: Lam et al. found developers who believed they’d fixed flaky tests by raising time values had often, when measured, changed nothing at all.
Re-Measure in 30 Days
One caveat: flakiness rate depends on where you measured, not only on the tests. In the same dataset, actions run from CI were flaky 0.8% of the time, scheduled batch runs 2.7%, and interactive debug runs 3.5%. Same tests, same applications. Record which context your baseline came from, or a suite that looks worse after a fix may only have been measured somewhere noisier.
Track three numbers: rerun rate, pull requests affected, and the share of investigated failures that turned out to be real. That third one is the honest signal. If it isn’t climbing, the audit hasn’t worked yet, and saying so beats declaring victory. If you’re building out QA metrics more broadly, flake rate belongs alongside build time and change failure rate, not in a separate report nobody opens.
The goal was never zero flaky tests. It’s a suite whose failures you believe.
Sources cited:
- Atlassian Engineering, Taming Test Flakiness (8 December 2025)
- Luo et al., An Empirical Analysis of Flaky Tests, FSE 2014: https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf
- Lam et al., ICSE 2020: https://doi.org/10.1145/3377811.3381749
Author
Karim Jouini, CEO at Thunders
Karim Jouini is the co-founder and CEO of Thunders, an AI-native testing platform whose agents write, run, and maintain test suites. He wrote and sold his first software at 16, studied software engineering and AI at INSA Toulouse and the University of Western Ontario, then spent seven years at Microsoft, where he first got close to testing and quality. In 2014 he co-founded Expensya, scaled it past 700,000 users across 100 countries, and sold it to Medius in 2023 for over $100 million. He founded Thunders in 2025 to rebuild software testing around AI agents rather than brittle scripts. He is a Forbes Technology Council member and one of France’s most active business angels, ranked fourth on the 2025 Angelsquare and Challenges list with investments in 26 startups.
Thunders were exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.
