AutomationSTAR

A 2-Day Test Automation Event

Track Talk T1

Apollo 13 and the Dangers of Automation Bias in Software Development

Andrew Brown

10:00-10:45, Tuesday 21st November

What does the Apollo 13 spacecraft disaster have in common with various automation problems within our DevOps, CI/CD, and agile developments?

The Apollo 13 disaster was caused in part by an overreliance upon automated data, a bias known as automation bias. This same bias causes us to make similar mistakes in software development, when we make decisions based upon the automated data from test suites and CI/CD systems.

In this session, I use the events leading up to the oxygen tank explosion on Apollo 13 to illustrate how automation bias led NASA engineers to not adequately monitor an automated system’s output, instead blindly accepting its suggestion that all was okay. We will explore what causes us to trust information from an automated system, and ignore contradictory information from a manual source, even when the manual source is more obviously correct.

We will then explore two major problems this bias causes within software development.

First, I will show how highly automated processes like CI/CD and DevOps cause software teams to base many development decisions upon data from automated systems, even when more reliable data from manual sources shows contradictory results. Second, you will see how the systems we deliver are increasingly used by senior decision-makers, who in turn trust that automated output, even when reliable manual data that obviously contradicts the automated output is readily accessible. I will then illustrate with an example from software development, where we explore how a set of unreliable automated tests within a DevOps CI/CD pipeline caused a development team to fail to spot an otherwise obvious error and also caused them to ignore critical warnings from the automated dashboard.

The talk will conclude by exploring the conditions present when we are most vulnerable to automation bias within software development, the reasons for that vulnerability, plus I’ll share several mitigation approaches that have been developed to reduce our vulnerability to this bias.

BACK TO PROGRAMME

What you will Learn

  1. Automation bias causes us to trust automated data, even when better data is available
  2. This bias will cause users of the systems we build to overly trust the data our systems provide
  3. We can mitigate these problems with specific techniques and careful design

Session Details

  • Introductory
  • 45 mins
  • Includes 15mins Q&A
Buy Conference Ticket

Track Talk T1

Apollo 13 and the Dangers of Automation Bias in Software Development

Session Speaker

Andrew Brown

Consultant - Expleo, UK

Dr Andrew Brown is a principal consultant at Expleo. He leads an independent line of research into understanding why we humans make the mistakes that lead to software defects. He has 25 years’ experience in the software industry. Previous roles include Head of QA at HMV, Head of QA at a financial software house and a test manager in Japan. He holds a degree in Physics and Maths, an MBA from Warwick Business School and a doctorate from Imperial College.