Gold Sponsor

The rapid growth in software delivery as a solution for all business needs for organizations has intensified the demand for efficient and comprehensive software testing processes. Ensuring the quality, reliability, and security of software products is of paramount importance as they become increasingly integrated into various aspects of our lives. Traditional software testing methods such as manual testing and automated test scripting, can no longer keep up with the responsibility to identify and rectify defects that make the delivered software ‘defect-free’. In addition to be viewed as inhibitors of fast-paced delivery,  they are viewed as high cost activities. 

The recent advancements in artificial intelligence (AI) and machine learning (ML) technologies have given rise to a new paradigm in software testing, known as Autonomous Test Generation (ATG). This approach leverages advanced algorithms and techniques to automatically generate relevant test cases, thereby reducing human intervention and enhancing the overall testing process.  This article is the beginning of a series related to Katalon’s implementation of autonomous test generation, where we will explore the different types of autonomous test generation and their benefits and limitations for the real-world Quality Engineers embedded in the teams of tomorrow. Our goal is to provide you with the knowledge and understanding necessary to make informed decisions about applying autonomous test generation solutions in your software testing process. 

Autonomous Test Generation – Definition and Key Concepts 

Autonomous test generation refers to the process of automatically identifying, creating and executing test cases for a software application, with minimal human intervention. This approach leverages AI and ML techniques to generate test cases that effectively identify feature defects, vulnerabilities, and user experience issues in software products. 

The Role of AI and ML in Autonomous Test Generation 

AI and ML technologies enable the analysis of large amounts of data associated with the usage patterns, and applied data characteristics during interactions to predict and generate a suite of evolving test scenarios necessary for maximum confidence of the application. By applying these algorithms, ATG can: 

1. Generate test cases that cover a strategic, data-centric range of scenarios. 

2. Adapt to evolving software requirements and deployed codebases. 

3. Optimize test cases based on historical data and prior outcomes. 

4. Confidently predict and generate new tests based on user observations. 

How does it compare to traditional methods? 

Compared to traditional manual and automated testing methods, ATG offers several advantages: 

1. Increased Confidence on system behaviors: ATG exhaustively tests different scenarios and edge cases, identifying potential bugs and errors, and providing more comprehensive test coverage. 

2. Efficiency in test creation: AI-powered test generation can quickly create test cases, reducing the time and effort required for testing. 

3. Effectiveness of workflow scenarios covered: Autonomous test generation can identify potential defects and vulnerabilities that might be missed by manual or scripted automated testing. 

4. Scalability in test coverage: The AI-driven approach can easily adapt to large and complex software systems, making it suitable for a wide range of applications and industries. 

5. Continuous adaptation to evolving usage patterns: Autonomous test generation can learn from testing results and adapt its strategies, leading to better test coverage and defect detection over time. 

Benefits of using autonomous test generation in software testing 

While manual testing remains an indispensable aspect of software development, its limitations can constrain the optimization of software quality. Autonomous test generation emerges as a groundbreaking solution, mitigating these challenges by automating the creation of test cases. Bringing an array of valuable advantages, autonomous test generation enhances time and cost efficiency, delivers extensive test coverage, and promotes seamless continuous testing and integration, thereby revolutionizing the software testing process. 

Time and cost efficiency 

By automating the process, developers can allocate their time to more value-added tasks, such as designing and implementing new features. Furthermore, the accelerated testing process reduces overall project costs by minimizing the need for human intervention and decreasing the likelihood of costly delays due to undetected issues. 

Comprehensive and accurate test coverage 

One of the most significant advantages of autonomous test generation is its ability to create a wide variety of test cases that account for a multitude of scenarios. This ensures more comprehensive test coverage compared to manual methods, reducing the likelihood of undiscovered defects or vulnerabilities in the software. Comprehensive test coverage is crucial for delivering high-quality software, as it increases the chances of identifying potential issues before they become critical. 

Continuous testing and integration for DevOps   

ATG enables continuous testing and integration, which is essential for modern software development methodologies like Agile and DevOps. With continuous testing, developers can identify and fix issues early in the development cycle, leading to faster delivery of high-quality software. This proactive approach to software testing also reduces the risk of costly post-release defects and contributes to an overall improvement in software quality. 

Maximizing Software Quality Through Autonomous Test Generation 

The true potential of ATG lies in its ability to not only streamline the software testing process but also elevate the overall quality of the final product. Autonomous test generation can lead to higher quality software by making an impact on defect detection, adaptability, and increased confidence in software releases. 

Increased awareness of defects  

By employing AI algorithms, ATG can generate a diverse range of test cases that account for complex scenarios. This breadth of test coverage increases the chances of continuously detecting defects that manual testing methods may overlook. Early identification and resolution of potential issues are crucial for maintaining high-quality standards in software development, making autonomous test generation an invaluable tool in the pursuit of high-quality digital experiences. 

Adaptability and scalability for evolving solutions 

One of the most significant advantages of autonomous test generation is its ability to adapt seamlessly to changes in software requirements or codebase updates. This adaptability ensures that test cases remain relevant and up-to-date, minimizing the risk of undetected defects due to obsolete test scenarios. Additionally, the scalability of autonomous test generation allows it to accommodate growing or evolving software projects, effectively addressing the quality assurance needs of increasingly complex systems. 

Increased confidence in software releases and positive user experiences  The comprehensive test coverage and efficient defect detection afforded by autonomous test generation foster a higher degree of confidence in the quality of software releases. This increased confidence translates into a reduced likelihood of encountering critical issues in production environments, leading to a better overall user experience. In turn, user satisfaction and trust in the software are enhanced, further solidifying the software’s reputation for quality and reliability. 

Challenges and limitations of autonomous test generation 

Challenge 1:  Incomplete Requirements and Specifications may result in limited tests  In many real-world situations, the documentation provided for software systems is often incomplete, ambiguous, or inconsistent, which can lead to inadequate test case generation. Autonomous test generation algorithms must be capable of handling these uncertainties, ideally filling in the gaps to create comprehensive test cases. This demands advanced AI techniques that can infer implicit information, which is still an area of ongoing research. 

Challenge 2:  Increased Test Counts based on Complexity of features may not be sustainable  As software systems become increasingly complex, the number of potential test cases grows exponentially. This creates a significant scalability challenge for autonomous test generation tools, as generating and executing exhaustive test cases can quickly become infeasible. Researchers are exploring various techniques, such as model-based testing, search-based testing, and AI-guided test generation, to overcome this obstacle. However, striking the right balance between coverage, complexity, and the computational resources required remains an open challenge. 

Challenge 3:  Integration with Existing Development Processes  Many organizations follow well-established development methodologies, such as Agile or DevOps, that dictate specific testing practices. Autonomous test generation tools need to seamlessly integrate with these processes to be effective, without disrupting the development team’s workflow. Additionally, compatibility with popular development and testing tools is essential to ensure the widespread adoption of autonomous test generation. 

Limitations  Despite the advancements in autonomous software test generation, certain limitations are inherent to the process. One such limitation is the inability to test certain aspects of software, such as usability, user experience, and aesthetics, which require human judgment. While autonomous test generation can produce a large number of test cases, it may not always produce meaningful or high-quality test cases that can effectively reveal defects. This highlights the need for continued research and development in the field to improve the overall quality and effectiveness of autonomously generated test cases.    

The importance of combining autonomous test generation with other testing techniques 

While autonomous software test generation offers the potential to reduce manual effort and enhance test coverage, it should not be considered a one-size-fits-all solution to all testing challenges. To achieve comprehensive and effective testing, it is crucial to combine autonomous test generation with other testing techniques, harnessing their complementary strengths to address various aspects of software quality.  

Complementing Test Coverage 

Autonomous test generation excels in creating test cases that cover a wide range of scenarios, based on the given requirements or specifications. However, as mentioned earlier, certain aspects of software quality, such as usability and user experience, require human judgment and cannot be effectively tested through autonomous test generation alone. By combining autonomous test generation with manual testing, organizations can ensure that both functional and non-functional requirements are adequately addressed.  Additionally, integration with other automated testing techniques, such as unit testing, integration testing, and system testing, can further bolster test coverage. While autonomous test generation can identify potential defects at a higher level, these other testing techniques can dive deeper into the code to catch issues that may not be apparent during higher-level testing. 

Leveraging Domain Knowledge and Expertise 

Incorporating domain knowledge and expertise is essential in the testing process, as it enables testers to create test cases that reflect real-world scenarios and potential edge cases. Autonomous test generation can struggle to capture this nuanced understanding of the software’s intended use. By combining autonomous test generation with domain-driven and expert-guided testing, organizations can ensure that test cases are not only comprehensive in terms of code coverage but also relevant and meaningful in terms of actual usage. 

Reducing Test Suite Maintenance Effort 

Test suite maintenance is an often-overlooked aspect of the testing process, as test cases must be updated and adapted to accommodate changes in the software’s requirements and functionality. While autonomous test generation can efficiently create new test cases, it may struggle to maintain and update existing test suites. By integrating autonomous test generation with techniques such as test case prioritization and regression testing, organizations can effectively manage their test suites, ensuring that they remain relevant and effective over time. 

Enhancing Quality (and maturity) of Testing Strategy 

Combining autonomous test generation with other testing techniques can lead to improved test case quality. Techniques such as mutation testing, which assesses the test suite’s ability to detect faults by injecting artificial defects into the code, can be employed to evaluate and enhance the quality of autonomously generated test cases. By iteratively refining the test cases, organizations can ensure that their test suites are not only comprehensive but also effective at detecting defects. 

Wrapping up 

Autonomous test generation, powered by AI and ML technologies, has the potential to revolutionize the software testing landscape, delivering significant benefits in terms of efficiency, effectiveness, and scalability. By automating the generation and execution of test cases, this approach promises to enhance software quality, foster a higher degree of confidence in software releases, and improve user satisfaction. 

However, it is important to acknowledge the challenges and limitations associated with autonomous test generation, such as handling incomplete requirements, addressing scalability and complexity, generating adequate test oracles, and integrating with existing development processes. Furthermore, autonomous test generation should not be considered a standalone solution; rather, it should be combined with other testing techniques to ensure comprehensive and effective testing that addresses both functional and non-functional requirements. 

As the software industry continues to evolve and grow, the need for efficient, effective, and scalable software testing methods will only increase. By embracing the opportunities and addressing the challenges presented by autonomous test generation, software developers, testers, and industry practitioners can leverage the power of AI-driven testing to deliver high-quality, reliable, and secure software products that meet the ever-changing demands of today’s complex digital world.  

Author

Alex Martins Vice President of Strategy at Katalon, Inc. 

Alex is a seasoned leader in the technology industry with extensive international business experience in agile software engineering, continuous testing, and DevOps. Starting out as a developer and then moving into software testing, Alex rose through the ranks to build and lead quality engineering practices across multiple enterprise companies from different industries around the world. Throughout his career, Alex has led the transformation of testing and quality assurance practices, as well as designing enhanced organizational structures to support the culture change necessary for successful adoption of modern software engineering approaches such as Agile and DevOps, and how these are being further transformed with Generative AI. 

Katalon is a Gold Sponsor at AutomationSTAR 2024, join us in Vienna.

Behind the pretty HTML cover of Allure Report is the idea that QA should be the responsibility of the entire team, not just QA – which means that test results should be accessible and readable by people without the QA or dev skill set. Report allows you to move past the details that don’t help you, staying at your preferred level of abstraction – and yet if you do need to drill into the code, it’s just a few mouse clicks away.

Report achieves this basic goal by being language-, framework-, and tool-agnostic. It can hide the peculiarities of your tech stack because it doesn’t depend upon it. So how does one become agnostic? You can’t do it through magic, you have to write tons of integrations, literally hundreds of thousands lines of code to integrate with anything and everything. Allure Report is a hub of integrations, and its structure is designed specifically with the purpose of making new integrations easier.

Let’s imagine that we’re writing a new integration for Report, and look at what resources we can leverage to make our job easier. We will be comparing how much effort we need to apply with Report and with other tools. We will start with the most straightforward advantages – the existing codebase; and then talk about more fundamental stuff like architecture and knowledge base.

Selenide native vs Selenide in Allure Report

To begin with, let us compare native reporting for Selenide with the way Selenide is integrated in Allure Report, and then see how difficult it was to write the integration for Report.

While creating simple reporting for Selenide is relatively easy, it’s a completely different story if you want to make quality test reports. In JUnit, there is only one extension point – the exception that is being thrown on test failure. You can jam the meta-information for the report into that exception, but working with this information will be difficult.

By default, Selenide and most other tools take the easy road. When Selenide reports on a failed test, what you get is just the text of the exception, a screenshot, and the HTML of the page at the time of failure:

If you’re the only tester on the project and all the tests are fresh in your memory, this might be more than enough – which is what the developers of Selenide are telling us.

Now, let’s compare this to Allure Report. If you run Report on a Selenide test with nothing plugged in, you’ll get just the text of the exception, same as with Selenide’s report.

But, as I’ve said before, the power of Allure Report is in its integrations. Things will change if we turn on allure-selenide and an integration for the framework you’re using (in this case – allure-junit). First (this is specific to the Selenide integration), we’re going to have to add the following line at the beginning of our test (or as a separate function with a @BeforeAll annotation):

SelenideLogger.addListener(“AllureSelenide”, new AllureSelenide());

Now, our test results have steps in them, and you can see precisely where the test has failed:

This can help you figure out why the test failed (whether the problem is in the test or in the code). You also get screenshots and the page source. Finally, with these integrations, you can wrap the function calls of your test inside the step() function or use the @Step annotation for functions you write yourself. This way, the steps displayed in test results will have custom names that you’ve written in human language, not lines of code. This makes the test results readable by people who don’t write Java (other testers, managers etc.). Adding all the steps might seem like a lot of extra work, but in the long run it actually saves time, because instead of answering a bunch of questions from other people in your company you can just direct them to test results written in plain English.

This is powerful stuff compared to what Selenide (and most other tools) offer as default reports. So here’s the main question for this article: how much effort did it take to achieve this? The source code for the allure-selenide integration is about 250 lines long. Considering the functionality that this provides, that’s almost nothing. Writing such an integration would probably be as easy as providing the bare exception that we get if we use Selenide’s native reporting.

This is the main takeaway: a proper integration with Allure Report takes about as much effort as a quick and easy integration with other tools (provided we’re talking about a language where Report has an established code base, such as Java or Python). How is that possible?

Common Libraries

The 250 lines of code in allure-selenide leverage files with about 500 lines of code from the allure-model section of allure-java, and about 1300 lines from allure-java-commons. These common libraries have been created to ease the process of making new integrations – and there are more than a dozen for Java alone that utilize these common libraries.

Writing these libraries is not a straightforward task. There are problems of execution here that can be extremely difficult to solve. For instance, when writing the allure-go integration, Anton Sinyaev spent several months solving the issue of parallel test execution (an issue which was left unsolved for 8 years in testify, the framework from which allure-go was forked). Such problems can be unique for a particular framework, which makes writing common libraries difficult. Generally speaking, once the process has been smoothed out, writing an integration for a framework like JUnit might take a month of work; but if there are no common libraries present, you could be looking at 4 or 5 months.

The JSON with the results

Let’s go deeper. What if we’re writing an integration for an entirely new language? Since the language is different, none of the code can be reused. Here, the example with Go is particularly telling, since it is quite unlike Java or Python, both in basic things like lack of classes, and in the way it works with threads. Because of this, not only was it not possible to reuse the code, but even the general solutions couldn’t be translated from one language to another. Then what HAS been reused in that case?

Arguably the most important part of Allure Report is its data format, the JSON file which stores the results of test runs. This is the meeting point for all languages, the thing that makes Allure Report language-agnostic. Designing that format took about a year, and it has incorporated some major architectural decisions – which means if you’re writing a new integration, you no longer have to think about this stuff. Thanks to this, the first, raw version of allure-go was written over a weekend – although it took several months to solve problems of execution and work out the kinks.

Experience

Finally, there is the least tangible asset of all – experience. Writing integrations is a peculiar field of programming, and a person skilled in it will be much more productive than someone who is just talented and generally experienced. If one had to guess, it would probably take 10 people about 2–3 years to re-do the work that’s been done on Allure Report, with one developer for each of the major languages and its common libraries, 2 or 3 devs for the reporter itself, an architect, and someone to work with the community.

Community

Allure Report’s community is not an asset strictly speaking, but when creating a new integration, it actually provides an extremely important role in several ways.

1. DEMAND. As we’ve already said, adding test reporting to a framework or a tool can take months of work if done properly. If you’re doing this purely for your own comfort, you’ll probably cut a lot of corners, do things quick and dirty. If, on the other hand, you’re working on something that is going to be used by millions of people, that’s motivation enough to sit around for an extra month or two and provide, say, proper parallel execution of tests.
2. EXPERIENCED DEVELOPERS. Here, we’re kind of returning to the previous section: the open-source nature of the tool allowed Qameta to get in touch with plenty of developers experienced in writing integrations, and hire from that pool.
3. THE INTEGRATIONS THEMSELVES. Allure report didn’t start out as a tool designed to integrate with anything and everything – the first version was just built for Junit 4 and Python. Pretty much everything outside allure-java and allure-python was initially developed outside Qameta, and then verified and internalized by the company.

All of this has been possible because there are many developers out there for whom Allure Report is a default tool – they are the bedrock of the community.

Conclusion

The structure of Allure Report didn’t appear all at once, like Athena did from the head of Zeus. It took many years of thinking, planning, and re-iterating on community feedback. What emerged as a result was a tool that was purpose-built to be extensible and to smooth out the creation of new integrations. Today, expanding upon this labor means leveraging the code, experience and architectural decisions that have been accumulated over the years.

If you’d like to learn more about Allure Report, we’ve recently created a dedicated site. Naturally, there’s documentation, as well as detailed info on all the integrations (under “Modules”). See if you can find your language and test framework there! And we’re planning to add much more stuff in the future, like guides, so don’t be a stranger and pay us a visit.

Author

Artem Eroshenko

CPO and Co-Founder of Qameta Software

Qameta Software are a Gold Sponsor at AutomationSTAR 20-21 Nov. 2023 in Berlin