Automated Guardrails for AI-generated Tests

There is a famous quote that says, “to err is human, but to really foul things up requires a computer”. This rings especially true when writing test automation code with help from AI Agents and copilots. This talk will cover concrete cases of how we can leverage both off-the-shelf and easy to make self-developed tools to ensure our test automation code and artifacts respect key quality constraints.

We will look simple tools for maintaining consistent spelling and the preventing maintenance nightmare of duplicated test code that should be resolved by refactoring. We will move on to ways of making custom tools that reject code we have identified as causing known issues.

We will integrate these tools with a model context provider so that our AI copilot learns from its own mistakes and generates test automation code that is consistent, reliable, non-repetitive and matches our repository’s standards.

Since the agents continually learn from our codebase, and work out how to write new tests by reading existing ones, succeeding with this endeavor will have a direct bearing on the quality of the code agents can produce over time.

If this process goes the wrong way, you will degrade the codebase with unreliable, hard to understand repetitive slop that will provide little inspiration for generating future, well-written new tests. Or, if you succeed in maintaining order and clarity, you can get effective and maintainable test automation at much higher velocity.