2026

How the right layers of abstraction improve collaboration, maintainability and technical flexibility

Open a test specification and you read: the customer places an order. Open the automation for the
same test and you find forty lines of selectors, waits and assertions. Both claim to describe one test.
Three sprints later a checkout step has moved, someone patched the code but not the specification,
and nobody can say which one is still true.

Anyone who has maintained a test suite for a year knows the situation. It isn’t a communication
problem. It’s a missing interface.

The people best placed to decide what to test are usually not the people who can automate it.
Business testers understand processes, requirements and the scenarios that matter. Automation
engineers understand frameworks, interfaces and the technical details of the system under test.
Traditional automation puts the two groups in a relay race: first specify, then automate. Every
change to the specification means a change to the code, and unless someone keeps paying the
manual cost of alignment, the two artifacts drift.

That drift was survivable when releases came quarterly. At weekly, it isn’t. Cycles get shorter,
expectations keep rising, and a specification nobody trusts stops being documentation and becomes
a liability.

More discipline won’t fix it. Abstraction will, applied in more places than most teams think.

Why Abstraction

Clean code has a rule for it: single level of abstraction. A function works on one level of detail instead
of mixing business logic with string handling. What you get is readability, understandability,
maintainability and reuse. Testing can borrow the principle, and two standards tell you where to put
it.

Abstraction 1: The Test Automation Architecture

ISTQB’s Generic Test Automation Architecture, laid out in the CTAL-TAE syllabus, applies the same
idea to the automation solution and separates four layers:

• Test generation — designing test cases, deciding what to test
• Test definition — test suites, test cases, test data, the reusable keyword library
• Test execution — the engine that runs the tests, logs and reports
• Test adaptation — the code that talks to the interfaces of the system under test

What makes the model useful is the separation it enforces. A change in one layer rarely forces a
change in another. A re-labelled button hits the adaptation layer, not two hundred test cases. Swap a UI-driven check for an API call on the same business step and the test definition layer never notices. The layers tell you where the boundaries belong, not how an intent stated at the top arrives as
executable code at the bottom.

Abstraction 2: The Test Specification

Keyword-driven testing is the answer. ISO/IEC/IEEE 29119-5 describes keyword-driven testing, and
the second edition from 2024 is explicit about hierarchy. At the bottom sit generic technical
keywords such as “Click” , “ Fill Text ” or “Get Element States ” . Those get composed
into business-level keywords: “ Log in as a registered customer ” , “ Place an
order ” . A test case is then a sequence of business keywords plus test data, readable by anyone
who knows the domain, with no code in sight.

The hierarchy itself isn’t the most interesting part. What it produces is a contract. The keyword name
and its parameters are the interface between the two worlds. A business tester can reorder steps,
add a data variant or build a new test case from existing keywords without touching automation
code. An automation engineer can rewrite whatever happens behind “ Place an order ” , new
UI, new API, new library, without invalidating a single test case. Each side changes what it owns. The
artifacts stay aligned because the thing that matters exists only once: the keyword.

Abstraction 3: The Tooling

Here is the step most teams skip. If the layers really are separate, no single tool must cover all of
them. That matters, because the two groups don’t want the same tool. Ask a business tester to
specify tests in VS Code and you lose them; an IDE isn’t their workplace and Git isn’t their
vocabulary. Ask an automation engineer to implement technical steps by clicking through a GUI and
you take away autocompletion, refactoring, version control and code review, which is most of what
makes engineering possible.

So choose per layer, then connect the layers. In our case, TestBench covers test definition, entirely
no-code: business testers maintain keywords and test data in a GUI and assemble test cases from
them. Robot Framework and VS Code cover the implementation, low-code where the existing Robot
Framework libraries suffice and real Python in user keywords where they don’t.
testbench2robotframework generates Robot Framework suites from a TestBench report and writes
the results back, and the TestBench extension for VS Code keeps keywords, descriptions and test
data in sync.

29119-5 expects this. The standard defines requirements for a common data exchange format so
that tools from different vendors can hand test cases, keywords and test data to each other.
Exchanging artifacts between tools isn’t a workaround for the all-in-one product nobody sells; the
standard treats it as the normal case.

Where It Breaks

Abstraction isn’t free. Keyword libraries grow, and an unmaintained catalogue of hundreds of nearidentical keywords is worse than none: nobody finds the right one, so everybody adds another.
Keywords cut too finely (Click button, Enter text) push technical detail straight back into the test
specification and cancel out the benefit. Layers hold only if someone owns the keyword library, if
naming conventions are agreed and enforced, and if keywords get reviewed as seriously as code.
Standards hand you a structure. Nobody hands you the discipline to keep it clean.

What You Actually Gain

Two things. A shared artifact that a business tester can read and an engineer can execute. And, less
obviously, a shared vocabulary. Domain-driven design calls this a ubiquitous language: one rigorous
set of terms used by domain experts and developers alike, in conversation and in the code. Keyword driven testing produces exactly that as a by-product. When “Place an order” means the same thing in a review meeting, in a test case and in a Robot Framework keyword, the two groups have stopped translating between worlds. They are editing the same sentence.

Further Reading

Leave Complexity Behind: No-Code Test Automation with Low-Code integration via Robot Framework and TestBench

Author

A flaky test passes and fails on the same code. Same commit, same environment, nothing changed. Run it ten times and you’ll get eight passes and two failures.

That inconsistency makes it worse than an ordinary failure. A real failure tells you something is broken. A flaky failure tells you nothing: you can’t say whether you found a bug or got unlucky. And yet most of us treat the two the same way: we shrug, we rerun, we move on.

Most teams know they have flaky tests. Far fewer know what they cost. Here is a way of finding out.

Why Flaky Tests Survive

Nobody keeps a flaky test on purpose. A build goes red, you recognise the usual suspect, you hit retry, it comes back green, you carry on with your day.

The problem is what those decisions add up to. Atlassian published figures in December 2025: flakiness was behind up to 21% of master build failures in their Jira Frontend repo, around 15% in the backend, and reruns cost more than 150,000 developer hours a year. Microsoft Research put the flaky share of their CI test failures at 13%.

Those are organisations with dedicated build infrastructure teams. If they’re bleeding that much time, your number is probably worse.

The Four Costs

Rerun cost. CI minutes spent on repeat runs, multiplied by your CI rate. It’s the easiest figure and usually the smallest, but calculate it first: a finance team understands it instantly, and it opens the door for the bigger numbers.

Investigation cost. Count the failures someone triaged that turned out not to be bugs. Multiply by average triage time, then by a loaded engineer hour. This normally lands at several times the rerun cost, and it’s the figure that moves a planning meeting.

Escaped defect exposure. Once a suite has cried wolf often enough, engineers rerun instead of investigating, and eventually a real bug rides through as noise. You can’t measure this cleanly, but you can count how many failures last quarter were closed without anyone ever opening them.

Lost trust. No formula for this one. It’s the day your team stops reading the test report. Name it anyway; it’s what makes the other three urgent.

Running The Audit

Two weeks is enough:

1. Turn on per-test run history. Most CI platforms log this by default.
2. For two weeks, tag every failure: real bug, flaky, or unknown.
3. Count reruns and pull requests affected per test.
4. Fill in the four costs above.
5. Rank by pull requests affected, not by how often a test fails.

In one week of platform data, ten organisations accounted for 82% of every flaky action recorded, and one for 32%. Flakiness concentrates, so your average tells you little about where to start.

A test that fails rarely but blocks your main branch every single time costs far more than one failing constantly in a suite nobody gates on. It’s the same instinct behind risk-based testing, just applied to maintenance rather than coverage.

What You’ll Find

Root causes cluster tightly. Analysing open-source Java projects, Luo et al. attributed roughly 45% of flaky tests to async wait, 20% to concurrency and 12% to test order dependency. The same paper found 54% of async-wait flakes were fixed by awaiting a response instead of a fixed timeout, and 74% of order-dependency flakes by cleaning shared state between runs.

That work underweights one category, because browsers have changed since 2014: selector drift. The test is logically correct, but the element it looks for has been renamed, moved or refactored away. Keep it separate: nothing is racing; it isn’t non-determinism at all. It’s maintenance debt in a reliability costume, and it responds to different treatment.

There’s a second thing that paper can’t see, because it isn’t a property of the test. One parameterised test appeared broadly unreliable until split by data fixture: Belgium failed 56% of runs, Portugal 50%, the Netherlands 40%, France 0%. Three fixtures had incomplete data, and the metric pooled them with the working ones. Before you classify a flaky test, check that it’s one test.

Fix, Delete, or Replace

Fix when the cause is genuine non-determinism in your own code: race conditions, shared state, a teardown that doesn’t mirror its setup. These are bugs in your tests, and they earn engineering time.

Delete when the test covers a path nobody uses, or duplicates coverage you already have. Most suites should be smaller, and deleting is a legitimate outcome of an audit, not an admission of failure.

Replace when the cause is selector drift. Repairing a brittle selector buys you about six weeks before the next one breaks. If a test falls over every time the interface is refactored, the fault lies in how it finds elements rather than in what it asserts, which is the point to look at test execution that adapts when the UI changes, rather than repairing locators by hand.

Quarantine buys time; it doesn’t fix anything, so treat it as a waiting room with a discharge date. And be suspicious of timeout adjustments: Lam et al. found developers who believed they’d fixed flaky tests by raising time values had often, when measured, changed nothing at all.

Re-Measure in 30 Days

One caveat: flakiness rate depends on where you measured, not only on the tests. In the same dataset, actions run from CI were flaky 0.8% of the time, scheduled batch runs 2.7%, and interactive debug runs 3.5%. Same tests, same applications. Record which context your baseline came from, or a suite that looks worse after a fix may only have been measured somewhere noisier.

Track three numbers: rerun rate, pull requests affected, and the share of investigated failures that turned out to be real. That third one is the honest signal. If it isn’t climbing, the audit hasn’t worked yet, and saying so beats declaring victory. If you’re building out QA metrics more broadly, flake rate belongs alongside build time and change failure rate, not in a separate report nobody opens.

The goal was never zero flaky tests. It’s a suite whose failures you believe.

Sources cited:

• Atlassian Engineering, Taming Test Flakiness (8 December 2025)
• Luo et al., An Empirical Analysis of Flaky Tests, FSE 2014: https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf
• Lam et al., ICSE 2020: https://doi.org/10.1145/3377811.3381749

Author

Thunders were exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.

AI coding agents can now build entire features from a single prompt. Tools like Claude Code, Codex CLI, and Cursor write code quickly and confidently. But there is a gap in this workflow that testers will recognize immediately: writing code and verifying that it works are two different disciplines.

The Verification Gap

Large language models are non-deterministic. The same prompt can produce different output on different runs, and agents occasionally hallucinate behavior that does not exist. One common workaround is asking the agent to review its own work, but self-verification in a loop is expensive and circular. An agent grading its own homework is not evidence.

What is missing is what testers have always insisted on: an independent, repeatable check against the real system. For web applications, that means opening an actual browser, executing the user journey step by step, and returning a clear pass or fail.

Closing the Gap from the Terminal

This is the problem Kane CLI, from TestMu AI, sets out to solve. It brings the KaneAI testing agent into the terminal, where both developers and coding agents already work. You describe a flow in natural language, such as a login, a form validation, or a checkout journey. Kane CLI runs it in a local Chrome browser and returns a deterministic result with a shareable evidence link containing video and a step trace.

The design choices reflect sound testing principles:

• Deterministic results. Every run ends in a binary pass or fail, backed by a real browser execution rather than the agent’s own opinion.
• Human-realistic actions. It only performs actions a real user could take inside the viewport. It never injects custom JavaScript to bypass validations and force a pass.
• Resilient, repeatable tests. Vision-based waiting detects loaders and animations, and self-healing absorbs cosmetic UI changes, so tests survive across runs and environments.
• Human-in-the-loop. When a flow hits an OTP or CAPTCHA, it pauses, asks a human to complete that step, and continues.

Where This Fits

For agents, verification happens through a structured agent mode: the coding agent calls Kane CLI, reads the machine-readable result, and decides whether to fix, continue, or escalate to a human. For teams, flows can export to native Playwright code and run headlessly in CI/CD pipelines.

The broader lesson holds regardless of tooling: as agents take on more of the building, independent validation becomes more important, not less. Verification, like testing itself, works best when it is decoupled from the thing being verified.

Author

TestMu AI are an AutomationSTAR 2026 exhibitor. Join us in Antwerp 4-5 Nov 2026.

Test automation is widely recognized as essential to modern delivery; it enables faster feedback, supports CI/CD practices, and increases release confidence. Yet in many organizations, automation growth lags behind development velocity. The reason is rarely a lack of intent.

It’s the effort required to convert validated manual tests into automation scripts.

Most teams already have structured manual test cases inside Jira. These tests represent validated functionality, documented business logic, and risk mitigation. However, the step that consistently slows progress is converting those structured manual tests into automation scripts compatible with frameworks like Playwright or Selenium. This often happens because automation engineers are in short supply within many teams. As a result, manual test cases need to be handed over, reviewed, and rewritten before they can be automated, creating a bottleneck in the overall testing process.

Xray’s AI Test Script Generation addresses this exact point of friction. Instead of treating manual validation and automation as separate tracks, it connects them directly within Jira, transforming validated test cases into framework-ready automation scripts while preserving traceability and workflow continuity.

Let’s explore how the feature works in practice, why it changes automation scalability, and how it fits into a broader AI-enabled quality strategy.

AI Test Script Generation operates on Test work items within Jira. For manual tests, the item must include at least one defined Test Step. For tests written in Gherkin format, the definition field must contain a valid scenario before automation can be generated. This ensures the AI has structured input to interpret the intended test behavior.

Why converting manual tests into automation slows teams down

Manual tests are rarely the problem; in fact, they are often well-structured and thoughtfully written. The challenge arises after validation, when those tests are handed off for rewriting in an automation framework. 

This rewrite phase introduces duplication, which means the logic has already been defined, but it must be translated step by step into executable code. That translation requires framework knowledge, coding expertise, and careful attention to application-specific details. When automation engineers are responsible for this conversion, their time is spent recreating what already exists in documented form.

Over time, this creates predictable consequences. Automation backlogs grow. Coverage expansion becomes dependent on a small group of specialists. The gap between manual validation and automated execution widens.

When scripts are generated, they may include placeholders such as  <BASE_URL> or <USERNAME>. These markers indicate values that should be replaced with environment-specific details, including application URLs, credentials, or element selectors, before the script is executed.

Xray’s AI Test Script Generation reduces that duplication of effort. It does not eliminate the need for expertise, but it shifts the starting point. Instead of beginning with a blank file, teams begin with a generated script that already reflects the structured intent of the manual test case. This fundamentally changes how quickly automation can scale.

How Xray’s AI Test Script Generation works step by step

The entire process is embedded inside Xray and Jira, which means teams do not need to introduce new platforms or adjust their existing test management structure.

The workflow begins with an existing structured manual test case. Because the feature relies on clearly defined steps and expected results, it builds directly on artifacts that already represent validated logic.

From there:

1. Open a manual test case in Xray
   The test case should include structured steps and expected outcomes.
   
2. Select “Automate Test”
   This option is available directly within the test case interface.
   
3. Choose your framework and language
   Select Playwright or Selenium and define the programming language used by your team. A version name is assigned to distinguish this automation instance.
   
4. Provide a short contextual prompt
   A brief description clarifies what the automated script should accomplish, giving the AI additional context.
   
5. Review generated outputs
   Sembi IQ generates:
   A framework-compatible automation script
   Configuration and dependency guidance
   A checklist of refinement tasks
   
6. Refine and iterate
   Before execution, teams adjust parameters such as URLs, environment variables, and element identifiers. The AI remains interactive, allowing additional modifications, such as testing invalid inputs or expanding validation logic.
   During this refinement phase, you can provide additional instructions to improve or modify the generated script. Each generation allows up to five iterative refinements. Once this limit is reached, a new script generation must be initiated.
   
7. Integrate and save
   The script can be downloaded or copied into an IDE. Finalizing the process creates a new automated test version inside Xray, maintaining traceability back to the original manual test and associated requirements.
   

The refinement stage is particularly important. Generated scripts are intentionally designed as structured starting points. Teams remain responsible for validation, optimization, and alignment with application-specific nuances before execution.

How this capability, powered by Sembi IQ, differs from generic AI code generators

Generic AI tools can produce automation snippets, but they lack structured context. They operate outside Jira. They do not understand requirement linkage. They do not preserve test version history or maintain traceability within a QA workflow.

Framework-native generators often assume coding proficiency and require scripts to be created independently from requirement artifacts. They typically start from scratch, detached from manual validation.

Xray’s AI Test Script Generation operates differently because it is embedded inside structured test management. It reads validated manual test steps directly from Jira, generates scripts within that context, and creates version-linked automated artifacts. This ensures continuity between requirements, manual tests, and automation.

The difference is not just in script generation. It is in maintaining governance and traceability throughout the transition from validation to automation.

Expanding automation participation without compromising control

Automation growth is frequently constrained by available engineering bandwidth. When every automated test requires deep framework expertise from the outset, coverage expansion becomes limited.

By generating the initial script structure, Xray’s AI Test Script Generation enables testers who understand functional intent to participate more directly in the automation process. Automation engineers can then focus on refining logic, improving resilience, and integrating scripts into CI pipelines rather than rewriting foundational steps.

This reduces repetitive scripting effort and shortens the path from validation to automation. It also lowers dependency on handoffs between roles, improving collaboration within QA and engineering teams.

Importantly, the feature maintains a human-in-the-loop approach. Generated scripts are editable, version-controlled, and traceable. Final execution remains under team oversight. The objective is acceleration with structure, not automation without control.

The impact at the organizational level includes faster automation velocity, improved coverage scalability, and stronger alignment between manual validation and automated execution.

Operational considerations

When using AI Test Script Generation, teams should also be aware of several practical constraints.

If files are uploaded during the generation process, their combined size cannot exceed 1 MB. The limit applies to the total size of all selected files. If the threshold is exceeded, the file selection must be adjusted before proceeding.

Supported file formats include TXT, MD, XML, YAML, PY, C, CPP, CC, C++, JAVA, JS, TS, JSX, TSX, SH, BASH, GO, RUST, SQL, HTML, RTF, CSV, JSON, DOC, DOCX, and PDF.

Recommended practices

As with any AI-assisted output, generated automation scripts should be treated as an initial foundation rather than a final deliverable.

To get the best results:

• Provide clear instructions that describe the intended test behavior.
  
• Keep prompts concise so the AI focuses on the relevant scenario.
  
• Always review and refine the generated script to ensure it aligns with your project’s automation standards and environment configuration.

How it fits into Xray’s broader AI ecosystem

Xray’s AI Test Script Generation builds on the AI foundation powered by Sembi IQ.

AI Test Case Generation helps teams accelerate the creation of structured test drafts from requirements across editions. AI Test Model Generation, available in Xray Enterprise, supports structured modeling of system behavior to strengthen coverage design.

Together, these capabilities create a connected workflow that moves from requirement interpretation to validation, behavioral modeling, and now directly into automation script generation.

Rather than existing as isolated AI features, they reinforce each stage of the testing lifecycle within Jira.

FAQ: Xray’s AI Test Script Generation

What is Xray’s AI Test Script Generation?

It is an AI-powered capability that transforms structured manual test cases into framework-ready automation scripts compatible with Playwright and Selenium, directly inside Jira.

Which Xray tiers include this feature?

Xray’s AI Test Script Generation is available for Xray Advanced and Xray Enterprise.

Does this replace automation engineers?

No. The feature accelerates initial script creation, but teams review, refine, and finalize scripts before execution. Expertise remains central to quality.

Is automation generated independently from requirements?

No. Scripts are generated from validated manual test cases that are already linked to requirements inside Jira, preserving context and traceability.

How does this relate to Xray’s other AI features?

It complements AI Test Case Generation and AI Test Model Generation by extending the AI-enabled workflow from requirement drafting and coverage modeling into executable automation.

---

Note on availability

We’ve transitioned our AI capabilities to “Enabled by Default” to ensure all users have immediate access to the productivity gains Sembi IQ provides. The platform’s full power is available out of the box, rather than hidden behind menus. However, your workflow comes first, and you have absolute control to toggle it off in the settings if it doesn’t fit your needs.

If you are still on Xray Cloud 6.16.0, your Jira admin must manually update Xray to the latest version in order to take advantage of all recent and future releases. (Including AI Test Script Generation)

Author

Welcome to the 17th edition of our World Quality Report (WQR)

Between the worlds we know and those still emerging, lies a horizon of possibility. Generative AI (Gen AI) and agentic technologies have brought us to this threshold—no longer distant concepts but forces actively reshaping how solutions are built, tested, and trusted.

Last year we explored the promise of Gen AI, automation, and human-in-the-loop systems; tools that became partners in testing and pushed the boundaries of Quality Engineering (QE). Since then, those possibilities have accelerated into reality, transforming how software is designed, developed, and delivered. Among all enterprise functions, QE stands out for its transformative potential.

Adapting to Emerging Worlds

This year’s theme, Adapting to Emerging Worlds, reflects the urgency of this shift. In a world where change is constant, adaptability has become the ultimate measure of resilience and leadership. For QE, this means rethinking roles, reimagining processes, and reshaping how quality itself is defined. The organizations that adapt fastest are the ones leading the next generation of smart, autonomous, and reliable engineering.

The 17th edition of the World Quality Report is your guide through this evolving terrain, equipping you with insights and tools to stay agile, anticipating the unexpected, and turning ambiguity into advantage. It goes beyond the headlines to answer some of the questions QE experts are asking:

• What’s working, and what’s not?
• How far has Gen AI really penetrated QE?
• How do we scale responsibly and effectively?
• What new skills and mindsets will define the future of QE?

Register now to receive your free copy of the World Quality Report 17th annual edition. Explore the data, learn from the leaders, and rethink how your organization can turn quality into a competitive advantage : https://www.opentext.com/resources/world-quality-report

Author

Opentext were exhibitors at AutomationSTAR Conference EXPO 2026. Join us in Antwerp 4-5th November 2026.