Honey, I Shrunk the Test Suite!

As testers we know that having ‘more tests’ isn’t automatically better. Tests have ongoing maintenance costs. Migrating a large test suite to a different technology or cleaning up legacy test suites are time-consuming tasks. Plus: ‘more tests’ also means slower feedback, because the whole suite now takes longer to execute!

At the same time, our gut feeling tells us that not all tests are equally valuable. Can’t we throw away the least valuable tests to save maintenance costs and execution time without sacrificing product quality? And then use the saved effort to create tests that do improve quality?

At PAYBACK we faced this problem when we had to migrate more than 1700 tests away from a legacy automation technology. Migrating all these tests would have taken us years. Manually sorting through them and trusting only our gut feeling was also not an option!

Fortunately, there’s a lot of research on this problem called ‘test suite minimization’: removing tests from a test suite while preserving its overall bug-finding power as best as possible. We applied the novel ‘AI Test Clustering’ approach that helped us to distinguish test cases that provide high value from those with little to no value.

In this talk, we will cover the theoretical fundamentals of this technique and show how we applied it to our own test suites. We will cover how we evaluated the approach using test coverage and how we ensured that important tests are not accidentally thrown away. We will discuss which limits of AI we found for our tasks and how we achieved high test coverage and a lot of confidence with only 20% of the original test suite. This saves us 37 person days of release test effort per year and over 400 person days of migration effort.