AutomationSTAR

Test Automation Conference Europe

EXPO

Business Process Assurance: Why Passing Tests Isn’t the Same as a Working Business

Business process assurance (BPA) validates whether an entire cross-system business workflow, such as order to cash or procure to pay, still works end to end after a change, rather than whether individual test scripts passed. It matters most in ERP environments like Dynamics 365, SAP, and Oracle Fusion, where upgrades, transports, and quarterly patches introduce risk on a fixed schedule.

Every business runs on processes that cross multiple systems: order to cash, procure to pay, hire to retire, record to report. When a system changes, whether it’s an ERP release wave, a cloud patch, or a new integration, the question that matters isn’t whether the individual test scripts passed. It’s whether the business process still works end to end, in production, under real conditions.

Business process assurance exists to answer that question. This guide covers what business process assurance (BPA) is, how it differs from testing individual applications, where manual approaches break down, and what a no-code, AI-assisted approach to business process assurance looks like for Dynamics 365, SAP, and Oracle Fusion environments in 2026.

Key Takeaways

  1. Business process assurance validates whether an entire cross-system workflow still works after a change, not just whether individual scripts passed. Passing tests and a working business process are not the same thing.
  2. Manual BPA does not scale against today’s release cadence. Dynamics 365 release upgrades, SAP transport changes, and Oracle Fusion quarterly patches all introduce risk on a schedule that manual regression cannot keep up with.
  3. Prioritize test cases that are high-frequency, business-critical, cross-system, and reasonably stable. Automating everything equally is not a strategy.
  4. No-code business process assurance lets IT directors, ERP project managers, functional consultants, and business analysts build and maintain automation directly, without depending on developers for every change.
  5. Business-Aware, Accountable AI extends BPA beyond execution speed. It understands process dependencies and business risk, and keeps every recommendation explainable and subject to human approval, so testing outcomes tie back to release confidence rather than pass or fail counts.

What Is Business Process Assurance?

Business process assurance validates that integrated systems and workflows function correctly together, not just individually. A typical enterprise business process depends on several components working in concert:

  • Applications: The ERP, CRM, and enterprise systems that run the process, such as Dynamics 365, SAP, or Oracle Fusion.
  • Databases: The systems storing the transactional and master data the process depends on.
  • User interfaces: The screens through which business users and functional teams interact with the process.
  • Integration points: The connections between systems, such as an ERP passing data to a CRM or a payroll provider.
  • Workflows: The sequence of steps, approvals, and handoffs that make up the end-to-end process.

In a software context, business process assurance looks at the same problem from inside the application layer: validating that modules, APIs, user interfaces, and databases interact correctly to support the business outcome the process was built for. The goal in both cases is the same. BPA confirms that a business-critical workflow, such as order to cash or procure to pay, still holds up end to end after a change, by simulating real operating conditions rather than testing each screen or module in isolation.

Why Business Process Assurance Is Different in ERP Environments

Generic business process assurance guidance treats every application the same way. In practice, the applications that carry the most business risk today are ERP systems, and each one introduces risk on its own schedule:

  • Microsoft Dynamics 365: Twice-yearly upgrades plus continuous updates mean Finance, Supply Chain, and HR processes are exposed to change constantly. Microsoft has confirmed that the Regression Suite Automation Tool (RSAT) reaches end of support on May 15, 2027, after which there will be no further bug fixes or maintenance. Teams still relying on RSAT for D365 regression coverage need a migration plan well before that date.
  • SAP ECC and S/4HANA: SAP’s mainstream maintenance for ECC 6.0 (Enhancement Packages 6 through 8) ends December 31, 2027; support for earlier enhancement packages (0 through 5) already ended in December 2025. Either deadline pushes organizations through S/4HANA migrations at the same time they need to keep Order-to-Cash and Procure-to-Pay processes running without disruption.
  • Oracle Fusion: Quarterly patch cycles mean Finance, HCM, and Supply Chain configurations are retested four times a year at minimum, with FlexField and configuration changes carrying real risk of silently corrupting financial reporting if they go unvalidated.

None of these deadlines are optional or negotiable on the vendor’s side. They are the reason business process assurance has moved from a periodic IT activity to a continuous, business-critical discipline.

Manual vs. Automated Business Process Assurance

AspectManual BPAAutomated BPA
Execution speedSlow, dependent on human effortFast, executes on demand
AccuracyProne to human errorConsistent, repeatable execution
Resource requirementRequires ongoing human resourcesRequires upfront setup, less ongoing effort
CostHigher ongoing labor costHigher initial investment, lower ongoing cost
ScalabilityDifficult to scale across company codes or business unitsScales through reusable, parameterized automation
MaintenanceManual and time-consumingSelf-healing automation adapts to UI and workflow changes
Business risk visibilitySurfaces after the fact, often after go-liveContinuous, flags risk before release

Common Business Process Assurance Challenges

BPA runs into the same obstacles in most enterprise environments:

  1. Complexity of business processes. A single process like Order to Cash can span CRM, ERP, logistics, and finance systems. Every integration point is a place where a change in one system can silently break a connected process.
  2. Resource constraints. Manual regression testing pulls functional consultants, ERP project managers, and business analysts away from higher-value work like exception handling and production readiness, every single release cycle.
  3. Constant change. Upgrades, transport changes, and quarterly patches mean test cases go stale quickly, and testing environments themselves can drift out of sync with production.

How to Build a Business Process Assurance Strategy

A working BPA strategy addresses these challenges directly:

  • Automate with intent, not everywhere. Prioritize test cases that are high-frequency, business-critical, cross-system, and relatively stable. A process that runs every sprint and touches revenue is worth automating first. A rarely used screen with a cosmetic UI change is not.
  • Validate end to end, not screen by screen. Cover the full path a transaction takes, from creation through every downstream system it touches, so a defect surfaces before it reaches production rather than after.
  • Build in regression testing as a standing practice. Every release, transport, or patch should trigger a regression pass against the business-critical processes it could affect, not a one-time test at go-live.
  • Design for reuse across company codes and business units. Parameterized automation that runs the same validated process against different datasets scales without multiplying the number of automation assets a team has to maintain.
  • Measure business risk, not just pass and fail counts. A dashboard showing 95% of tests passed doesn’t tell you whether the 5% that failed sits inside a revenue-critical workflow. Tie testing outcomes back to which business processes are actually at risk.

What to Look for in Business Process Assurance Software

Choosing the right platform comes down to a few criteria that matter in ERP environments specifically:

  1. No-code test creation. IT directors, ERP project managers, functional consultants, and business analysts should be able to build and maintain automation directly, without writing code or waiting on a developer for every change.
  2. Business-Aware, Accountable AI. AI that understands business process dependencies and prioritizes validation based on business risk, not just screen coverage, with every recommendation explainable and subject to human approval.
  3. End-to-end test coverage. The ability to validate a process from start to finish across every system it touches, not just the primary application.
  4. Cross-platform compatibility. Testing across web, desktop, ERP, and mainframe environments, spanning a wide range of technologies from a single platform.
  5. Parallel and distributed execution. Running tests concurrently to cut regression cycle time, especially valuable when the same process needs to be validated across multiple company codes or business units.
  6. CI/CD integration. Fitting into existing DevOps pipelines so testing runs continuously as changes are deployed, rather than as a separate manual gate.

How Avo Assure Approaches Business Process Assurance

Avo Assure goes beyond test automation to Enterprise Business Process Assurance. With no-code test creation, Business-Aware, Accountable AI, and end-to-end test coverage, Avo Assure helps teams validate business processes across Dynamics 365, SAP, Oracle, and every connected system, without relying on developers.

In practice, that looks like Synergy Marine Group, one of the world’s largest ship management companies, validating standardized SAP business processes across multiple company codes and connected applications, including Microsoft Dynamics 365 Business Central and SAP Concur. Working with Accenture, the team reduced manual regression effort by 90% and reached approximately 80% automation coverage of planned business scenarios, using reusable automation instead of building separate scripts for every company code.

Read the full Synergy Marine Group and Accenture case study for the complete breakdown of the engagement and its results.

Across enterprise deployments generally, Avo Assure customers report:

  • 10x faster time to market with intelligent, end-to-end test automation
  • 90%+ business process automation coverage
  • 85% less maintenance effort, through self-healing automation
  • 80% reduction in effort required to create and execute tests

The right business process assurance approach determines whether a release goes smoothly or becomes a production incident.

See how Avo Assure helps you move beyond test automation to business process assurance and book a demo to get started.

Frequently Asked Questions:

What is business process assurance?

Business process assurance (BPA) validates that an integrated business workflow, such as order to cash or procure to pay, works correctly end to end across every system it touches, rather than testing each application or screen in isolation.

How is business process assurance different from regular software testing?

Regular software testing typically validates a single application or module. Business process assurance validates the full workflow a business depends on, which often spans multiple systems, such as an ERP, a CRM, and a payroll provider, and confirms the process still functions correctly as a whole after a change.

Why does business process assurance matter more in ERP environments?

ERP systems like Dynamics 365, SAP, and Oracle Fusion change on a fixed vendor schedule, whether through upgrades, transport changes, or quarterly patches. Each change carries a real chance of breaking a business-critical process like financial close or order fulfillment, which makes continuous validation a business necessity rather than a periodic IT task.

Can business processes assurance be automated without a dedicated QA team?

Yes. No-code platforms let functional consultants, business analysts, and ERP project managers build and maintain business process test directly, without relying on developers or ongoing support from implementation partners.

What is the difference between test automation and business process assurance?

Test automation confirms that a script ran and passed. Business process assurance goes further: it confirms that the underlying business process, such as Order to Cash, still works correctly end to end after a change, and ties that outcome to business risk rather than a pass or fail count alone.

How often should business-critical processes be regression tested?

Business-critical processes should be regression tested every release cycle, and again after any configuration, transport, or patch change, since ERP environments on continuous release schedules introduce risk multiple times a year, not just at major upgrades.

Does business process assurance cover processes that span multiple ERP or enterprise systems?

Yes. Processes like Order to Cash or Hire to Retire rarely stay inside a single system. Business process assurance should validate the full cross-system path, including CRM, logistics, finance, and HCM platforms connected to the core ERP.

What should I look for in business process assurance software?

Look for no-code test creation, Business-Aware AI that prioritizes validation by business risk rather than screen count, end-to-end coverage across every system in the process, and the ability to reuse automation across company codes or business units without rebuilding it each time.

Avo Automation are Exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

Who Tests the Tests? The Hidden Risk in AI-Generated Test Cases

A test suite goes green across the board. Every check passes, and the release ships to production. Then the support tickets start coming in: a regression got through, something the tests were supposed to catch. Nobody investigates too deeply, since it happened only once and could easily have been an isolated case.

Then it happens again. The same pattern repeats: a green suite, a real bug, and a gap nobody can explain. That is when teams finally start asking the question they should have asked from the beginning: did these tests ever actually work, or did they simply never fail?

That question matters more now than it did two years ago because AI can write test cases in seconds, allowing teams to generate large numbers of tests in a fraction of the time. The problem is that teams can quickly end up with a suite full of tests that look thorough but fail to catch the bugs that matter. A green checkmark tells you that a test passed, but it says nothing about whether that test was ever capable of catching a real bug.

The AI doesn’t know your product, it’s guessing

An AI model writing a test case doesn’t know your product. It knows patterns. It has seen thousands of checkout flows, login forms, and API calls, and when you ask it to test yours, it draws on what those examples have in common. That’s often a reasonable starting point, since the basic structure of a checkout test is fairly predictable across products. But looking like a good test and actually checking the right thing are not the same, and the model has no way to tell the difference on its own.

Left unchecked, the model guesses. It might assert that a page loaded, that a field is not empty, that a button exists, none of which tell you the feature actually works the way your business needs it to. The test runs, returns green, and sits in your suite as false confidence.

Does this test actually earn its green checkmark?

Take a concrete case: an AI tool generates a test for a checkout discount code. The test enters a code, clicks apply, and checks that the page didn’t show an error. It passes. Should you trust it?

Here’s what actually you should do:

Break the code on purpose: this is one of the most reliable checks you can run, so it deserves close attention. Go into the discount logic and deliberately break it: make the discount apply twice, ignore the expiry date, or apply to an already-discounted item. If the test still passes after you’ve broken the logic it is meant to protect, it was never checking the right thing. This is called mutation testing. Instead of asking, “Does the test pass?”, it asks, “Can this test fail when it should?” Gartner recommends this approach for teams using AI to expand test coverage, since AI-generated tests can lack the depth needed to catch real defects on their own. In practice, you create several broken versions of the discount logic and check that the test catches each one. Any failure to detect a mutation tells you exactly what needs to be fixed before you can trust the green result.

Have a second AI model check the assertion: Once a test survives mutation testing, ask a different question: does its assertion actually match the requirement, or does it just confirm that something happened? For the discount code, the requirement isn’t simply that no error appears. It’s that the discount matches what the code entitles the customer to and that the order total reflects it correctly. A second model can compare the assertion against the written requirement and flag any mismatch. The goal is to close the gap between a test that confirms something happened and one that confirms the right thing happened.

Check for near-duplicates: AI tools generate tests quickly, but that can also mean generating the same test several times with small variations. If you already have four tests confirming that a discount code works correctly, a fifth that only changes the code string is not adding coverage. It is just adding another test to maintain. The ASTQB calls this “false confidence from volume”: a large test suite can look thorough while still leaving the same risks uncovered. Merge tests that overlap heavily and keep near-duplicates only when they cover a genuinely different path, such as combining a discount code with a gift card.

Save human review for what matters most: Not every test needs the same level of scrutiny. Sonar’s 2026 State of Code Developer Survey found that 96% of developers don’t fully trust AI-generated code to be functionally correct, while only 48% always check it before committing. The same caution should apply to AI-generated tests, particularly in areas where a failure can have a direct impact on customers and the business. Pricing, payments, and authentication deserve closer attention: a discount applied twice, a payment processed more than once, or a login that gives access to the wrong session can all have serious consequences. Reserve manual review for these high-stakes flows, while using mutation testing and AI-assisted review to validate the rest of the suite.

The next question testingON is built to answer

Most AI test-generation tools, including the ones enterprise QA teams are piloting right now, stop at generation. They write the test, hand it back green, and move on. They don’t run the checks described above, they don’t break the code on purpose, they don’t flag near-duplicates, they don’t know which paths are high-stakes enough to need a human. That’s true of most AI test-generation vendors too: speed is the pitch, and validation is left entirely to whoever’s using the tool.

We built testingON’s roadmap around closing that specific gap. Under an initiative we’re calling Agentic AI, we’re building an agent whose job is exactly what this post describes: taking a generated test and verifying it can actually catch a real defect, not just checking that it runs green. It sits alongside a companion agent that generates tests overnight against new user stories, and another that runs tests in the background and reports on execution results and performance, so generation, validation, and execution work as one connected process instead of three separate

tools bolted together. This is a roadmap direction we’re actively building toward, not a shipped feature today, but it’s a direct answer to the exact problem this post lays out.

If you’re piloting AI test generation and want to talk about how to validate what it’s producing before it reaches production, see what testingON is building.

Author

Tiago Honorato, Quality Management DevOps & Automation Associate Director at Noesis

I am a technology leader with more than 18 years of experience in Software Engineering, Quality Assurance, Product Innovation, and Digital Transformation.

I started my career as a Software Developer at Nokia Siemens Networks after completing my Master’s degree in Computer Science in Lisbon, Portugal. Shortly after, I embraced a new challenge at Noesis, where I founded and scaled the company’s Test Automation practice, helping clients accelerate their software delivery lifecycle and significantly improve quality standards.

Although I transitioned from development to Quality Assurance, I have always maintained a strong connection to software engineering — combining deep technical expertise with strategic quality vision. Over time, I led multiple initiatives across test automation, application development, and quality transformation programs for major enterprise clients.

Leveraging this cross-functional experience, I conceived and led the creation of testingON — a test management and automation platform designed to reduce testing time while increasing coverage and enabling non-technical users to actively contribute to quality processes. The platform integrates with leading test management, CI/CD, and project management tools and is currently used by major Telco, Retail, Banking, and Insurance organizations in Portugal and Brazil.

I later accepted a new challenge at NTT DATA, where I served as Senior Manager responsible for the Quality Assurance & Testing offering, leading strategic accounts, strengthening delivery capabilities, and expanding quality services.

Subsequently, I returned to Noesis as Associate Director to lead the strategic evolution of testingON. In this role, I am responsible for product vision, innovation roadmap, and growth strategy — including the integration of Generative AI capabilities and the modernization of the platform’s architecture to ensure scalability, cloud readiness, and long-term competitiveness.

testingON are Platinum Partners at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

Meet BQA and Teammaite at AutomationSTAR: from testing expertise to AI-powered quality

Visit BQA and Teammaite at AutomationSTAR and discover how testing expertise, AI and automation come together – with AI as the accelerator and the professional in the driver-seat.

AI-powered development is delivering code faster, sometimes producing an entire feature in a single iteration. Testers therefore need to analyse specifications, risks, test strategy, coverage and automation just as quickly – without sacrificing quality.

That raises a practical question:

How do you stay relevant, valuable and effective as quality becomes more AI-driven?

At AutomationSTAR, BQA and Teammaite will explore that question with testers, automation engineers, QA leads and business owners. Our message is simple: AI should not replace testing expertise; it should make that expertise more powerful.

Quality Starts Before Testing Starts

Requirements often look clear until testing reveals the gaps: exception flows are undefined, acceptance criteria are not measurable, risks are unclear or the business expectation does not match what was built.

BQA therefore believes quality must move upstream. Requirements, user stories and acceptance criteria should be analysed before development starts, with risks and test ideas linked to business intent from the beginning.

With AI support, Teammaite can analyse requirements for ambiguity, missing information, inconsistencies and testability, helping teams build a stronger foundation for test design and automation.

The challenge is not simply generating tests faster, but deciding what should be tested:

  • What does the specification really mean?
  • Are the requirements complete, unambiguous and testable?
  • Where are the biggest product and business risks?
  • Which test techniques and level of coverage are appropriate?
  • How do we translate those choices into existing processes and tools in a traceable way?

From Dpecification to the Right Test Coverage

At our stand, we will demonstrate one recognisable end-to-end quality flow:

Specifications -> requirements -> quality improvement -> risk analysis -> test strategy -> test design -> execution and management

Teammaite can process documents, models, tables or existing tickets, while a BQA consultant determines the appropriate context and extraction approach.

Teammaite then identifies ambiguity, inconsistencies, missing information and testability issues. AI can propose product risks, impact and priorities, but consultants and stakeholders validate the business context and risk weighting.

Based on clearer requirements and validated risks, teams can define relevant test levels, test types, techniques and desired coverage.

The goal is not automatically more tests, but targeted and explainable coverage of the risks that matter.

From Tester to AI Test Orchestrator

AI accelerates the process, but it does not remove the need for testing expertise. Experienced testers understand context, recognise risk, judge whether test cases are meaningful and decide what should be automated or explored by people.

BQA calls this role the AI test orchestrator: a quality professional who uses AI deliberately, critically and effectively across the software development lifecycle. For test managers and QA leads, governance, prioritisation, traceability and release confidence remain essential.

The key question is no longer only:

“Can AI create tests?”

It is:

“Can we explain why these tests cover the right risks?”

That is the central question behind our stand at AutomationSTAR.

AI in Control: People Make the Decisions

Responsible AI must be visible in the way quality decisions are made: AI makes proposals; people make decisions.

Important steps can include review and approval. Sources, assumptions, modifications and decisions remain traceable, uncertainty is explicit, and the degree of autonomy can be adjusted to the organisation and the risk involved.

Teammaite delivers the intelligent, integrated workflow. BQA provides the human link – from selecting the approach and facilitating stakeholder discussions to risk analysis, test strategy, governance and implementation.

Connect AI to the Tools you Already Use

AI-supported quality should not become an isolated experiment. Approved results need to fit the organisation’s existing way of working.

The approach can connect with tools such as Robot Framework, Testersuite, Jira and Azure DevOps while maintaining traceability from specification through requirements and tests to findings.

Together, BQA and Teammaite support the full quality flow: requirements analysis, test strategy, test cases, automation and release insight.

Teammaite brings the AI capability; BQA brings experienced consultants who understand testing, test management, automation, governance and software delivery in complex organisations.

AI tooling alone does not change an organisation. People, process and implementation determine whether AI becomes a useful colleague or just another experiment.

Bring Us Your Specification

At AutomationSTAR, we want to go beyond a generic AI product presentation.

Bring a specification or sample requirement and take the spec-to-strategy challenge.

In a few minutes, Teammaite can highlight potential improvements, risks and suggested test coverage. A BQA consultant will discuss which proposals make sense, which assumptions need challenging, where human judgement is required and what integration into your quality process could look like.

You leave with a compact AI-in-control quick scan and a clearer view of what AI-supported quality could mean for your organisation.

Come and Challenge Us

We believe in practical quality improvement, not AI hype for its own sake.

Come to the BQA and Teammaite stand with a testing challenge, requirements problem, automation bottleneck, management concern or simply one existing specification.

Ask us what AI can do, where human expertise remains essential and how your team can move from specification to improved requirements, validated risks, an appropriate test strategy and integrated test facts.

And come prepared for one question from us:

Can you explain why your current tests cover the right risks?

Authors:

Sander Ulrich – Teammaite

Gerben Braakman – BQA       

Frank van der Kuur – BQA

BQA are exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

The Keyword Is the Contract

How the right layers of abstraction improve collaboration, maintainability and technical flexibility

Open a test specification and you read: the customer places an order. Open the automation for the
same test and you find forty lines of selectors, waits and assertions. Both claim to describe one test.
Three sprints later a checkout step has moved, someone patched the code but not the specification,
and nobody can say which one is still true.

Anyone who has maintained a test suite for a year knows the situation. It isn’t a communication
problem. It’s a missing interface.

The people best placed to decide what to test are usually not the people who can automate it.
Business testers understand processes, requirements and the scenarios that matter. Automation
engineers understand frameworks, interfaces and the technical details of the system under test.
Traditional automation puts the two groups in a relay race: first specify, then automate. Every
change to the specification means a change to the code, and unless someone keeps paying the
manual cost of alignment, the two artifacts drift.

That drift was survivable when releases came quarterly. At weekly, it isn’t. Cycles get shorter,
expectations keep rising, and a specification nobody trusts stops being documentation and becomes
a liability.

More discipline won’t fix it. Abstraction will, applied in more places than most teams think.

Why Abstraction

Clean code has a rule for it: single level of abstraction. A function works on one level of detail instead
of mixing business logic with string handling. What you get is readability, understandability,
maintainability and reuse. Testing can borrow the principle, and two standards tell you where to put
it.

Abstraction 1: The Test Automation Architecture

ISTQB’s Generic Test Automation Architecture, laid out in the CTAL-TAE syllabus, applies the same
idea to the automation solution and separates four layers:

  • Test generation — designing test cases, deciding what to test
  • Test definition — test suites, test cases, test data, the reusable keyword library
  • Test execution — the engine that runs the tests, logs and reports
  • Test adaptation — the code that talks to the interfaces of the system under test

What makes the model useful is the separation it enforces. A change in one layer rarely forces a
change in another. A re-labelled button hits the adaptation layer, not two hundred test cases. Swap a UI-driven check for an API call on the same business step and the test definition layer never notices. The layers tell you where the boundaries belong, not how an intent stated at the top arrives as
executable code at the bottom.

Abstraction 2: The Test Specification

Keyword-driven testing is the answer. ISO/IEC/IEEE 29119-5 describes keyword-driven testing, and
the second edition from 2024 is explicit about hierarchy. At the bottom sit generic technical
keywords such as “Click” , “ Fill Text ” or “Get Element States ” . Those get composed
into business-level keywords: “ Log in as a registered customer ” , “ Place an
order ” . A test case is then a sequence of business keywords plus test data, readable by anyone
who knows the domain, with no code in sight.

The hierarchy itself isn’t the most interesting part. What it produces is a contract. The keyword name
and its parameters are the interface between the two worlds. A business tester can reorder steps,
add a data variant or build a new test case from existing keywords without touching automation
code. An automation engineer can rewrite whatever happens behind “ Place an order ” , new
UI, new API, new library, without invalidating a single test case. Each side changes what it owns. The
artifacts stay aligned because the thing that matters exists only once: the keyword.

Abstraction 3: The Tooling

Here is the step most teams skip. If the layers really are separate, no single tool must cover all of
them. That matters, because the two groups don’t want the same tool. Ask a business tester to
specify tests in VS Code and you lose them; an IDE isn’t their workplace and Git isn’t their
vocabulary. Ask an automation engineer to implement technical steps by clicking through a GUI and
you take away autocompletion, refactoring, version control and code review, which is most of what
makes engineering possible.


So choose per layer, then connect the layers. In our case, TestBench covers test definition, entirely
no-code: business testers maintain keywords and test data in a GUI and assemble test cases from
them. Robot Framework and VS Code cover the implementation, low-code where the existing Robot
Framework libraries suffice and real Python in user keywords where they don’t.
testbench2robotframework generates Robot Framework suites from a TestBench report and writes
the results back, and the TestBench extension for VS Code keeps keywords, descriptions and test
data in sync.

29119-5 expects this. The standard defines requirements for a common data exchange format so
that tools from different vendors can hand test cases, keywords and test data to each other.
Exchanging artifacts between tools isn’t a workaround for the all-in-one product nobody sells; the
standard treats it as the normal case.

Where It Breaks

Abstraction isn’t free. Keyword libraries grow, and an unmaintained catalogue of hundreds of nearidentical keywords is worse than none: nobody finds the right one, so everybody adds another.
Keywords cut too finely (Click button, Enter text) push technical detail straight back into the test
specification and cancel out the benefit. Layers hold only if someone owns the keyword library, if
naming conventions are agreed and enforced, and if keywords get reviewed as seriously as code.
Standards hand you a structure. Nobody hands you the discipline to keep it clean.

What You Actually Gain

Two things. A shared artifact that a business tester can read and an engineer can execute. And, less
obviously, a shared vocabulary. Domain-driven design calls this a ubiquitous language: one rigorous
set of terms used by domain experts and developers alike, in conversation and in the code. Keyword driven testing produces exactly that as a by-product. When “Place an order” means the same thing in a review meeting, in a test case and in a Robot Framework keyword, the two groups have stopped translating between worlds. They are editing the same sentence.

Further Reading

Leave Complexity Behind: No-Code Test Automation with Low-Code integration via Robot Framework and TestBench

Author

Falk Altrock, Product Owner TestBench
imbus AG

TestBench are exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

The Flaky Test Audit: What Your Suite Really Costs You

A flaky test passes and fails on the same code. Same commit, same environment, nothing changed. Run it ten times and you’ll get eight passes and two failures.

That inconsistency makes it worse than an ordinary failure. A real failure tells you something is broken. A flaky failure tells you nothing: you can’t say whether you found a bug or got unlucky. And yet most of us treat the two the same way: we shrug, we rerun, we move on.

Most teams know they have flaky tests. Far fewer know what they cost. Here is a way of finding out.

Why Flaky Tests Survive

Nobody keeps a flaky test on purpose. A build goes red, you recognise the usual suspect, you hit retry, it comes back green, you carry on with your day.

The problem is what those decisions add up to. Atlassian published figures in December 2025: flakiness was behind up to 21% of master build failures in their Jira Frontend repo, around 15% in the backend, and reruns cost more than 150,000 developer hours a year. Microsoft Research put the flaky share of their CI test failures at 13%.

Those are organisations with dedicated build infrastructure teams. If they’re bleeding that much time, your number is probably worse.

The Four Costs

Rerun cost. CI minutes spent on repeat runs, multiplied by your CI rate. It’s the easiest figure and usually the smallest, but calculate it first: a finance team understands it instantly, and it opens the door for the bigger numbers.

Investigation cost. Count the failures someone triaged that turned out not to be bugs. Multiply by average triage time, then by a loaded engineer hour. This normally lands at several times the rerun cost, and it’s the figure that moves a planning meeting.

Escaped defect exposure. Once a suite has cried wolf often enough, engineers rerun instead of investigating, and eventually a real bug rides through as noise. You can’t measure this cleanly, but you can count how many failures last quarter were closed without anyone ever opening them.

Lost trust. No formula for this one. It’s the day your team stops reading the test report. Name it anyway; it’s what makes the other three urgent.

Running The Audit

Two weeks is enough:

  1. Turn on per-test run history. Most CI platforms log this by default.
  2. For two weeks, tag every failure: real bug, flaky, or unknown.
  3. Count reruns and pull requests affected per test.
  4. Fill in the four costs above.
  5. Rank by pull requests affected, not by how often a test fails.

In one week of platform data, ten organisations accounted for 82% of every flaky action recorded, and one for 32%. Flakiness concentrates, so your average tells you little about where to start.

A test that fails rarely but blocks your main branch every single time costs far more than one failing constantly in a suite nobody gates on. It’s the same instinct behind risk-based testing, just applied to maintenance rather than coverage.

What You’ll Find

Root causes cluster tightly. Analysing open-source Java projects, Luo et al. attributed roughly 45% of flaky tests to async wait, 20% to concurrency and 12% to test order dependency. The same paper found 54% of async-wait flakes were fixed by awaiting a response instead of a fixed timeout, and 74% of order-dependency flakes by cleaning shared state between runs.

That work underweights one category, because browsers have changed since 2014: selector drift. The test is logically correct, but the element it looks for has been renamed, moved or refactored away. Keep it separate: nothing is racing; it isn’t non-determinism at all. It’s maintenance debt in a reliability costume, and it responds to different treatment.

There’s a second thing that paper can’t see, because it isn’t a property of the test. One parameterised test appeared broadly unreliable until split by data fixture: Belgium failed 56% of runs, Portugal 50%, the Netherlands 40%, France 0%. Three fixtures had incomplete data, and the metric pooled them with the working ones. Before you classify a flaky test, check that it’s one test.

Fix, Delete, or Replace

Fix when the cause is genuine non-determinism in your own code: race conditions, shared state, a teardown that doesn’t mirror its setup. These are bugs in your tests, and they earn engineering time.

Delete when the test covers a path nobody uses, or duplicates coverage you already have. Most suites should be smaller, and deleting is a legitimate outcome of an audit, not an admission of failure.

Replace when the cause is selector drift. Repairing a brittle selector buys you about six weeks before the next one breaks. If a test falls over every time the interface is refactored, the fault lies in how it finds elements rather than in what it asserts, which is the point to look at test execution that adapts when the UI changes, rather than repairing locators by hand.

Quarantine buys time; it doesn’t fix anything, so treat it as a waiting room with a discharge date. And be suspicious of timeout adjustments: Lam et al. found developers who believed they’d fixed flaky tests by raising time values had often, when measured, changed nothing at all.

Re-Measure in 30 Days

One caveat: flakiness rate depends on where you measured, not only on the tests. In the same dataset, actions run from CI were flaky 0.8% of the time, scheduled batch runs 2.7%, and interactive debug runs 3.5%. Same tests, same applications. Record which context your baseline came from, or a suite that looks worse after a fix may only have been measured somewhere noisier.

Track three numbers: rerun rate, pull requests affected, and the share of investigated failures that turned out to be real. That third one is the honest signal. If it isn’t climbing, the audit hasn’t worked yet, and saying so beats declaring victory. If you’re building out QA metrics more broadly, flake rate belongs alongside build time and change failure rate, not in a separate report nobody opens.

The goal was never zero flaky tests. It’s a suite whose failures you believe.

Sources cited:

Author

Karim Jouini, CEO at Thunders

Karim Jouini is the co-founder and CEO of Thunders, an AI-native testing platform whose agents write, run, and maintain test suites. He wrote and sold his first software at 16, studied software engineering and AI at INSA Toulouse and the University of Western Ontario, then spent seven years at Microsoft, where he first got close to testing and quality. In 2014 he co-founded Expensya, scaled it past 700,000 users across 100 countries, and sold it to Medius in 2023 for over $100 million. He founded Thunders in 2025 to rebuild software testing around AI agents rather than brittle scripts. He is a Forbes Technology Council member and one of France’s most active business angels, ranked fourth on the 2025 Angelsquare and Challenges list with investments in 26 startups.

Thunders were exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

part of the