Track Talk T5

Honey, I Shrunk the Test Suite!

Sebastian Mörtl

Fabian Streitel

11:00-11:45 CET, Thursday 5th November

As testers we know that having ‘more tests’ isn’t automatically better. Tests have ongoing maintenance costs. Migrating a large test suite to a different technology or cleaning up legacy test suites are time-consuming tasks. Plus: ‘more tests’ also means slower feedback, because the whole suite now takes longer to execute!

At the same time, our gut feeling tells us that not all tests are equally valuable. Can’t we throw away the least valuable tests to save maintenance costs and execution time without sacrificing product quality? And then use the saved effort to create tests that do improve quality?

At PAYBACK we faced this problem when we had to migrate more than 1700 tests away from a legacy automation technology. Migrating all these tests would have taken us years. Manually sorting through them and trusting only our gut feeling was also not an option!

Fortunately, there’s a lot of research on this problem called ‘test suite minimization’: removing tests from a test suite while preserving its overall bug-finding power as best as possible. We applied the novel ‘AI Test Clustering’ approach that helped us to distinguish test cases that provide high value from those with little to no value.

In this talk, we will cover the theoretical fundamentals of this technique and show how we applied it to our own test suites. We will cover how we evaluated the approach using test coverage and how we ensured that important tests are not accidentally thrown away. We will discuss which limits of AI we found for our tasks and how we achieved high test coverage and a lot of confidence with only 20% of the original test suite. This saves us 37 person days of release test effort per year and over 400 person days of migration effort.

BACK TO PROGRAMME

What you will Learn

Systematically removing redundancy from large end-to-end test suites yields much smaller test suites with almost the same effectiveness.
AI test clustering can detect redundant test cases that cause maintenance costs and provide little value.
Where can you rely on an AI-based approach and what are its limitations that require an expert in the loop?

Session Details

Intermediate
30 minutes
15mins Q&A
Automation Strategy & Architecture

Buy Conference Ticket

Session Speakers

Sebastian Mörtl

Test Architect - PAYBACK GmbH, Germany

Sebastian Mörtl is a Test Architect and leader of the QA Guild at PAYBACK in Munich. His career began with a degree in landscape architecture. In 2000 he transitioned to IT as a software consultant, and since 2009 has gathered extensive experience across various agile environments as Product Owner, Developer, Test Engineer, and Test Architect. Sebastian is deeply passionate about comprehensive test automation and specifically about automating quality gates throughout the software delivery lifecycle.

Co Speaker

Fabian Streitel

Software Quality Expert - CQSE, Germany

Fabian Streitel studied computer science at the Technical University of Munich and has been working as a software quality expert at CQSE for over 10 years. Over the years, Fabian has observed the testing processes of hundreds of companies and has helped to improve them. He leads the test intelligence research at CQSE.