The Evolution of Arrange-Act-Assert

For decades, testers have been using assertions and patterns like ‘arrange, act, assert’ to implement test automation, and those patterns have served us really well. Now, with the advent of non-deterministic AI applications, our testing needs to evolve too.

Just a few years ago, testers could get almost everything required for test automation using crisp logic like ‘does the answer equal 8’ and ‘do you see the correct value in the grid?’.

Today’s AI systems are a different kind of challenge for testers. This session will take a deep dive into validation patterns for non-deterministic AI systems. Instead of implementing specific assertions, we will explore using a ‘judge LLM’ to collect meaningful observations that are indicators of quality.

Conceptually, this is very similar to the notion of a traditional rubric that can generate a meaningful quality score. An AI judge evaluates each response against a set of criteria, producing structured observations such as pass/fail decisions, confidence levels, and justification.

Those observations are collected as data, not verdicts, enabling trend analysis, regression detection, and evidence-based release decisions. This presentation will not be focused on syntax, but it will be a code- and data-walkthrough engineering talk. We’ll focus on execution flow, evaluation models, and how this approach fits current systems. Attendees will see how rubric-driven evaluation bridges the gap between traditional testing and non-deterministic quality assessments.