EXPO

A test suite goes green across the board. Every check passes, and the release ships to production. Then the support tickets start coming in: a regression got through, something the tests were supposed to catch. Nobody investigates too deeply, since it happened only once and could easily have been an isolated case.

Then it happens again. The same pattern repeats: a green suite, a real bug, and a gap nobody can explain. That is when teams finally start asking the question they should have asked from the beginning: did these tests ever actually work, or did they simply never fail?

That question matters more now than it did two years ago because AI can write test cases in seconds, allowing teams to generate large numbers of tests in a fraction of the time. The problem is that teams can quickly end up with a suite full of tests that look thorough but fail to catch the bugs that matter. A green checkmark tells you that a test passed, but it says nothing about whether that test was ever capable of catching a real bug.

The AI doesn’t know your product, it’s guessing

An AI model writing a test case doesn’t know your product. It knows patterns. It has seen thousands of checkout flows, login forms, and API calls, and when you ask it to test yours, it draws on what those examples have in common. That’s often a reasonable starting point, since the basic structure of a checkout test is fairly predictable across products. But looking like a good test and actually checking the right thing are not the same, and the model has no way to tell the difference on its own.

Left unchecked, the model guesses. It might assert that a page loaded, that a field is not empty, that a button exists, none of which tell you the feature actually works the way your business needs it to. The test runs, returns green, and sits in your suite as false confidence.

Does this test actually earn its green checkmark?

Take a concrete case: an AI tool generates a test for a checkout discount code. The test enters a code, clicks apply, and checks that the page didn’t show an error. It passes. Should you trust it?

Here’s what actually you should do:

Break the code on purpose: this is one of the most reliable checks you can run, so it deserves close attention. Go into the discount logic and deliberately break it: make the discount apply twice, ignore the expiry date, or apply to an already-discounted item. If the test still passes after you’ve broken the logic it is meant to protect, it was never checking the right thing. This is called mutation testing. Instead of asking, “Does the test pass?”, it asks, “Can this test fail when it should?” Gartner recommends this approach for teams using AI to expand test coverage, since AI-generated tests can lack the depth needed to catch real defects on their own. In practice, you create several broken versions of the discount logic and check that the test catches each one. Any failure to detect a mutation tells you exactly what needs to be fixed before you can trust the green result.

Have a second AI model check the assertion: Once a test survives mutation testing, ask a different question: does its assertion actually match the requirement, or does it just confirm that something happened? For the discount code, the requirement isn’t simply that no error appears. It’s that the discount matches what the code entitles the customer to and that the order total reflects it correctly. A second model can compare the assertion against the written requirement and flag any mismatch. The goal is to close the gap between a test that confirms something happened and one that confirms the right thing happened.

Check for near-duplicates: AI tools generate tests quickly, but that can also mean generating the same test several times with small variations. If you already have four tests confirming that a discount code works correctly, a fifth that only changes the code string is not adding coverage. It is just adding another test to maintain. The ASTQB calls this “false confidence from volume”: a large test suite can look thorough while still leaving the same risks uncovered. Merge tests that overlap heavily and keep near-duplicates only when they cover a genuinely different path, such as combining a discount code with a gift card.

Save human review for what matters most: Not every test needs the same level of scrutiny. Sonar’s 2026 State of Code Developer Survey found that 96% of developers don’t fully trust AI-generated code to be functionally correct, while only 48% always check it before committing. The same caution should apply to AI-generated tests, particularly in areas where a failure can have a direct impact on customers and the business. Pricing, payments, and authentication deserve closer attention: a discount applied twice, a payment processed more than once, or a login that gives access to the wrong session can all have serious consequences. Reserve manual review for these high-stakes flows, while using mutation testing and AI-assisted review to validate the rest of the suite.

The next question testingON is built to answer

Most AI test-generation tools, including the ones enterprise QA teams are piloting right now, stop at generation. They write the test, hand it back green, and move on. They don’t run the checks described above, they don’t break the code on purpose, they don’t flag near-duplicates, they don’t know which paths are high-stakes enough to need a human. That’s true of most AI test-generation vendors too: speed is the pitch, and validation is left entirely to whoever’s using the tool.

We built testingON’s roadmap around closing that specific gap. Under an initiative we’re calling Agentic AI, we’re building an agent whose job is exactly what this post describes: taking a generated test and verifying it can actually catch a real defect, not just checking that it runs green. It sits alongside a companion agent that generates tests overnight against new user stories, and another that runs tests in the background and reports on execution results and performance, so generation, validation, and execution work as one connected process instead of three separate

tools bolted together. This is a roadmap direction we’re actively building toward, not a shipped feature today, but it’s a direct answer to the exact problem this post lays out.

If you’re piloting AI test generation and want to talk about how to validate what it’s producing before it reaches production, see what testingON is building.

Author

testingON are Platinum Partners at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

Visit BQA and Teammaite at AutomationSTAR and discover how testing expertise, AI and automation come together – with AI as the accelerator and the professional in the driver-seat.

AI-powered development is delivering code faster, sometimes producing an entire feature in a single iteration. Testers therefore need to analyse specifications, risks, test strategy, coverage and automation just as quickly – without sacrificing quality.

That raises a practical question:

How do you stay relevant, valuable and effective as quality becomes more AI-driven?

At AutomationSTAR, BQA and Teammaite will explore that question with testers, automation engineers, QA leads and business owners. Our message is simple: AI should not replace testing expertise; it should make that expertise more powerful.

Quality Starts Before Testing Starts

Requirements often look clear until testing reveals the gaps: exception flows are undefined, acceptance criteria are not measurable, risks are unclear or the business expectation does not match what was built.

BQA therefore believes quality must move upstream. Requirements, user stories and acceptance criteria should be analysed before development starts, with risks and test ideas linked to business intent from the beginning.

With AI support, Teammaite can analyse requirements for ambiguity, missing information, inconsistencies and testability, helping teams build a stronger foundation for test design and automation.

The challenge is not simply generating tests faster, but deciding what should be tested:

• What does the specification really mean?
• Are the requirements complete, unambiguous and testable?
• Where are the biggest product and business risks?
• Which test techniques and level of coverage are appropriate?
• How do we translate those choices into existing processes and tools in a traceable way?

From Dpecification to the Right Test Coverage

At our stand, we will demonstrate one recognisable end-to-end quality flow:

Specifications -> requirements -> quality improvement -> risk analysis -> test strategy -> test design -> execution and management

Teammaite can process documents, models, tables or existing tickets, while a BQA consultant determines the appropriate context and extraction approach.

Teammaite then identifies ambiguity, inconsistencies, missing information and testability issues. AI can propose product risks, impact and priorities, but consultants and stakeholders validate the business context and risk weighting.

Based on clearer requirements and validated risks, teams can define relevant test levels, test types, techniques and desired coverage.

The goal is not automatically more tests, but targeted and explainable coverage of the risks that matter.

From Tester to AI Test Orchestrator

AI accelerates the process, but it does not remove the need for testing expertise. Experienced testers understand context, recognise risk, judge whether test cases are meaningful and decide what should be automated or explored by people.

BQA calls this role the AI test orchestrator: a quality professional who uses AI deliberately, critically and effectively across the software development lifecycle. For test managers and QA leads, governance, prioritisation, traceability and release confidence remain essential.

The key question is no longer only:

“Can AI create tests?”

It is:

“Can we explain why these tests cover the right risks?”

That is the central question behind our stand at AutomationSTAR.

AI in Control: People Make the Decisions

Responsible AI must be visible in the way quality decisions are made: AI makes proposals; people make decisions.

Important steps can include review and approval. Sources, assumptions, modifications and decisions remain traceable, uncertainty is explicit, and the degree of autonomy can be adjusted to the organisation and the risk involved.

Teammaite delivers the intelligent, integrated workflow. BQA provides the human link – from selecting the approach and facilitating stakeholder discussions to risk analysis, test strategy, governance and implementation.

Connect AI to the Tools you Already Use

AI-supported quality should not become an isolated experiment. Approved results need to fit the organisation’s existing way of working.

The approach can connect with tools such as Robot Framework, Testersuite, Jira and Azure DevOps while maintaining traceability from specification through requirements and tests to findings.

Together, BQA and Teammaite support the full quality flow: requirements analysis, test strategy, test cases, automation and release insight.

Teammaite brings the AI capability; BQA brings experienced consultants who understand testing, test management, automation, governance and software delivery in complex organisations.

AI tooling alone does not change an organisation. People, process and implementation determine whether AI becomes a useful colleague or just another experiment.

Bring Us Your Specification

At AutomationSTAR, we want to go beyond a generic AI product presentation.

Bring a specification or sample requirement and take the spec-to-strategy challenge.

In a few minutes, Teammaite can highlight potential improvements, risks and suggested test coverage. A BQA consultant will discuss which proposals make sense, which assumptions need challenging, where human judgement is required and what integration into your quality process could look like.

You leave with a compact AI-in-control quick scan and a clearer view of what AI-supported quality could mean for your organisation.

Come and Challenge Us

We believe in practical quality improvement, not AI hype for its own sake.

Come to the BQA and Teammaite stand with a testing challenge, requirements problem, automation bottleneck, management concern or simply one existing specification.

Ask us what AI can do, where human expertise remains essential and how your team can move from specification to improved requirements, validated risks, an appropriate test strategy and integrated test facts.

And come prepared for one question from us:

Can you explain why your current tests cover the right risks?

Authors:

BQA are exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

How the right layers of abstraction improve collaboration, maintainability and technical flexibility

Open a test specification and you read: the customer places an order. Open the automation for the
same test and you find forty lines of selectors, waits and assertions. Both claim to describe one test.
Three sprints later a checkout step has moved, someone patched the code but not the specification,
and nobody can say which one is still true.

Anyone who has maintained a test suite for a year knows the situation. It isn’t a communication
problem. It’s a missing interface.

The people best placed to decide what to test are usually not the people who can automate it.
Business testers understand processes, requirements and the scenarios that matter. Automation
engineers understand frameworks, interfaces and the technical details of the system under test.
Traditional automation puts the two groups in a relay race: first specify, then automate. Every
change to the specification means a change to the code, and unless someone keeps paying the
manual cost of alignment, the two artifacts drift.

That drift was survivable when releases came quarterly. At weekly, it isn’t. Cycles get shorter,
expectations keep rising, and a specification nobody trusts stops being documentation and becomes
a liability.

More discipline won’t fix it. Abstraction will, applied in more places than most teams think.

Why Abstraction

Clean code has a rule for it: single level of abstraction. A function works on one level of detail instead
of mixing business logic with string handling. What you get is readability, understandability,
maintainability and reuse. Testing can borrow the principle, and two standards tell you where to put
it.

Abstraction 1: The Test Automation Architecture

ISTQB’s Generic Test Automation Architecture, laid out in the CTAL-TAE syllabus, applies the same
idea to the automation solution and separates four layers:

• Test generation — designing test cases, deciding what to test
• Test definition — test suites, test cases, test data, the reusable keyword library
• Test execution — the engine that runs the tests, logs and reports
• Test adaptation — the code that talks to the interfaces of the system under test

What makes the model useful is the separation it enforces. A change in one layer rarely forces a
change in another. A re-labelled button hits the adaptation layer, not two hundred test cases. Swap a UI-driven check for an API call on the same business step and the test definition layer never notices. The layers tell you where the boundaries belong, not how an intent stated at the top arrives as
executable code at the bottom.

Abstraction 2: The Test Specification

Keyword-driven testing is the answer. ISO/IEC/IEEE 29119-5 describes keyword-driven testing, and
the second edition from 2024 is explicit about hierarchy. At the bottom sit generic technical
keywords such as “Click” , “ Fill Text ” or “Get Element States ” . Those get composed
into business-level keywords: “ Log in as a registered customer ” , “ Place an
order ” . A test case is then a sequence of business keywords plus test data, readable by anyone
who knows the domain, with no code in sight.

The hierarchy itself isn’t the most interesting part. What it produces is a contract. The keyword name
and its parameters are the interface between the two worlds. A business tester can reorder steps,
add a data variant or build a new test case from existing keywords without touching automation
code. An automation engineer can rewrite whatever happens behind “ Place an order ” , new
UI, new API, new library, without invalidating a single test case. Each side changes what it owns. The
artifacts stay aligned because the thing that matters exists only once: the keyword.

Abstraction 3: The Tooling

Here is the step most teams skip. If the layers really are separate, no single tool must cover all of
them. That matters, because the two groups don’t want the same tool. Ask a business tester to
specify tests in VS Code and you lose them; an IDE isn’t their workplace and Git isn’t their
vocabulary. Ask an automation engineer to implement technical steps by clicking through a GUI and
you take away autocompletion, refactoring, version control and code review, which is most of what
makes engineering possible.

So choose per layer, then connect the layers. In our case, TestBench covers test definition, entirely
no-code: business testers maintain keywords and test data in a GUI and assemble test cases from
them. Robot Framework and VS Code cover the implementation, low-code where the existing Robot
Framework libraries suffice and real Python in user keywords where they don’t.
testbench2robotframework generates Robot Framework suites from a TestBench report and writes
the results back, and the TestBench extension for VS Code keeps keywords, descriptions and test
data in sync.

29119-5 expects this. The standard defines requirements for a common data exchange format so
that tools from different vendors can hand test cases, keywords and test data to each other.
Exchanging artifacts between tools isn’t a workaround for the all-in-one product nobody sells; the
standard treats it as the normal case.

Where It Breaks

Abstraction isn’t free. Keyword libraries grow, and an unmaintained catalogue of hundreds of nearidentical keywords is worse than none: nobody finds the right one, so everybody adds another.
Keywords cut too finely (Click button, Enter text) push technical detail straight back into the test
specification and cancel out the benefit. Layers hold only if someone owns the keyword library, if
naming conventions are agreed and enforced, and if keywords get reviewed as seriously as code.
Standards hand you a structure. Nobody hands you the discipline to keep it clean.

What You Actually Gain

Two things. A shared artifact that a business tester can read and an engineer can execute. And, less
obviously, a shared vocabulary. Domain-driven design calls this a ubiquitous language: one rigorous
set of terms used by domain experts and developers alike, in conversation and in the code. Keyword driven testing produces exactly that as a by-product. When “Place an order” means the same thing in a review meeting, in a test case and in a Robot Framework keyword, the two groups have stopped translating between worlds. They are editing the same sentence.

Further Reading

Leave Complexity Behind: No-Code Test Automation with Low-Code integration via Robot Framework and TestBench

Author

A flaky test passes and fails on the same code. Same commit, same environment, nothing changed. Run it ten times and you’ll get eight passes and two failures.

That inconsistency makes it worse than an ordinary failure. A real failure tells you something is broken. A flaky failure tells you nothing: you can’t say whether you found a bug or got unlucky. And yet most of us treat the two the same way: we shrug, we rerun, we move on.

Most teams know they have flaky tests. Far fewer know what they cost. Here is a way of finding out.

Why Flaky Tests Survive

Nobody keeps a flaky test on purpose. A build goes red, you recognise the usual suspect, you hit retry, it comes back green, you carry on with your day.

The problem is what those decisions add up to. Atlassian published figures in December 2025: flakiness was behind up to 21% of master build failures in their Jira Frontend repo, around 15% in the backend, and reruns cost more than 150,000 developer hours a year. Microsoft Research put the flaky share of their CI test failures at 13%.

Those are organisations with dedicated build infrastructure teams. If they’re bleeding that much time, your number is probably worse.

The Four Costs

Rerun cost. CI minutes spent on repeat runs, multiplied by your CI rate. It’s the easiest figure and usually the smallest, but calculate it first: a finance team understands it instantly, and it opens the door for the bigger numbers.

Investigation cost. Count the failures someone triaged that turned out not to be bugs. Multiply by average triage time, then by a loaded engineer hour. This normally lands at several times the rerun cost, and it’s the figure that moves a planning meeting.

Escaped defect exposure. Once a suite has cried wolf often enough, engineers rerun instead of investigating, and eventually a real bug rides through as noise. You can’t measure this cleanly, but you can count how many failures last quarter were closed without anyone ever opening them.

Lost trust. No formula for this one. It’s the day your team stops reading the test report. Name it anyway; it’s what makes the other three urgent.

Running The Audit

Two weeks is enough:

1. Turn on per-test run history. Most CI platforms log this by default.
2. For two weeks, tag every failure: real bug, flaky, or unknown.
3. Count reruns and pull requests affected per test.
4. Fill in the four costs above.
5. Rank by pull requests affected, not by how often a test fails.

In one week of platform data, ten organisations accounted for 82% of every flaky action recorded, and one for 32%. Flakiness concentrates, so your average tells you little about where to start.

A test that fails rarely but blocks your main branch every single time costs far more than one failing constantly in a suite nobody gates on. It’s the same instinct behind risk-based testing, just applied to maintenance rather than coverage.

What You’ll Find

Root causes cluster tightly. Analysing open-source Java projects, Luo et al. attributed roughly 45% of flaky tests to async wait, 20% to concurrency and 12% to test order dependency. The same paper found 54% of async-wait flakes were fixed by awaiting a response instead of a fixed timeout, and 74% of order-dependency flakes by cleaning shared state between runs.

That work underweights one category, because browsers have changed since 2014: selector drift. The test is logically correct, but the element it looks for has been renamed, moved or refactored away. Keep it separate: nothing is racing; it isn’t non-determinism at all. It’s maintenance debt in a reliability costume, and it responds to different treatment.

There’s a second thing that paper can’t see, because it isn’t a property of the test. One parameterised test appeared broadly unreliable until split by data fixture: Belgium failed 56% of runs, Portugal 50%, the Netherlands 40%, France 0%. Three fixtures had incomplete data, and the metric pooled them with the working ones. Before you classify a flaky test, check that it’s one test.

Fix, Delete, or Replace

Fix when the cause is genuine non-determinism in your own code: race conditions, shared state, a teardown that doesn’t mirror its setup. These are bugs in your tests, and they earn engineering time.

Delete when the test covers a path nobody uses, or duplicates coverage you already have. Most suites should be smaller, and deleting is a legitimate outcome of an audit, not an admission of failure.

Replace when the cause is selector drift. Repairing a brittle selector buys you about six weeks before the next one breaks. If a test falls over every time the interface is refactored, the fault lies in how it finds elements rather than in what it asserts, which is the point to look at test execution that adapts when the UI changes, rather than repairing locators by hand.

Quarantine buys time; it doesn’t fix anything, so treat it as a waiting room with a discharge date. And be suspicious of timeout adjustments: Lam et al. found developers who believed they’d fixed flaky tests by raising time values had often, when measured, changed nothing at all.

Re-Measure in 30 Days

One caveat: flakiness rate depends on where you measured, not only on the tests. In the same dataset, actions run from CI were flaky 0.8% of the time, scheduled batch runs 2.7%, and interactive debug runs 3.5%. Same tests, same applications. Record which context your baseline came from, or a suite that looks worse after a fix may only have been measured somewhere noisier.

Track three numbers: rerun rate, pull requests affected, and the share of investigated failures that turned out to be real. That third one is the honest signal. If it isn’t climbing, the audit hasn’t worked yet, and saying so beats declaring victory. If you’re building out QA metrics more broadly, flake rate belongs alongside build time and change failure rate, not in a separate report nobody opens.

The goal was never zero flaky tests. It’s a suite whose failures you believe.

Sources cited:

• Atlassian Engineering, Taming Test Flakiness (8 December 2025)
• Luo et al., An Empirical Analysis of Flaky Tests, FSE 2014: https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf
• Lam et al., ICSE 2020: https://doi.org/10.1145/3377811.3381749

Author

Thunders were exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

AI coding agents can now build entire features from a single prompt. Tools like Claude Code, Codex CLI, and Cursor write code quickly and confidently. But there is a gap in this workflow that testers will recognize immediately: writing code and verifying that it works are two different disciplines.

The Verification Gap

Large language models are non-deterministic. The same prompt can produce different output on different runs, and agents occasionally hallucinate behavior that does not exist. One common workaround is asking the agent to review its own work, but self-verification in a loop is expensive and circular. An agent grading its own homework is not evidence.

What is missing is what testers have always insisted on: an independent, repeatable check against the real system. For web applications, that means opening an actual browser, executing the user journey step by step, and returning a clear pass or fail.

Closing the Gap from the Terminal

This is the problem Kane CLI, from TestMu AI, sets out to solve. It brings the KaneAI testing agent into the terminal, where both developers and coding agents already work. You describe a flow in natural language, such as a login, a form validation, or a checkout journey. Kane CLI runs it in a local Chrome browser and returns a deterministic result with a shareable evidence link containing video and a step trace.

The design choices reflect sound testing principles:

• Deterministic results. Every run ends in a binary pass or fail, backed by a real browser execution rather than the agent’s own opinion.
• Human-realistic actions. It only performs actions a real user could take inside the viewport. It never injects custom JavaScript to bypass validations and force a pass.
• Resilient, repeatable tests. Vision-based waiting detects loaders and animations, and self-healing absorbs cosmetic UI changes, so tests survive across runs and environments.
• Human-in-the-loop. When a flow hits an OTP or CAPTCHA, it pauses, asks a human to complete that step, and continues.

Where This Fits

For agents, verification happens through a structured agent mode: the coding agent calls Kane CLI, reads the machine-readable result, and decides whether to fix, continue, or escalate to a human. For teams, flows can export to native Playwright code and run headlessly in CI/CD pipelines.

The broader lesson holds regardless of tooling: as agents take on more of the building, independent validation becomes more important, not less. Verification, like testing itself, works best when it is decoupled from the thing being verified.

Author

TestMu AI are an AutomationSTAR 2026 exhibitor. Join us in Antwerp 4-5 Nov 2026.