Sustainable Execution: Which Tests Can We Skip This Time?

As our software grows, we face the challenge of testing an increasing amount of functionality. This leads to more and more automated tests. Many teams I collaborate with manage thousands, or even hundreds of thousands, of automated tests. To identify bugs early, these tests are executed frequently, often many times each day. Typically, all of them are executed in every single test run.

While this approach reduces feedback times and helps in early bug detection, it comes with significant energy costs: the more tests we execute, the greater the energy consumption. Research indicates that the annual energy usage for test execution in a medium-sized project is comparable to that of a European household.

We can do better: The more frequently we run our tests, the less beneficial each individual test execution becomes, as the likelihood of uncovering new bugs diminishes. For instance, if no change was made to the code a test executes since a test’s last execution, it cannot find a new bug. Therefore, it is unnecessary to execute all the tests all the time.

Test selection approaches automatically identify those tests that have the highest probability of discovering new bugs at any given time. By running these tests more frequently than the other tests, we can save a lot of energy and still find the bugs in our software.

For the past decade, I’ve devoted my work to test selection in both research and practical applications. Our team has developed and tested various test selection strategies within our own development processes and those of our clients. In this talk, I will share our successes and setbacks, and discuss the true impact of these strategies on making test execution more sustainable.