How to Automate Visual Testing on a Freeform Canvas UI?

Problem:
With the many advancements in Visual Testing, automated web-based UI verification can be performed quickly and accurately. However, web-apps that render their UIs entirely inside a Canvas element are left out. The reason is that Canvas-based apps cannot provide a reliable reference snapshot of the UI. Without a reliable visual expectation, the automation has no ground-truths to verify against. This forces the QA team to rely on manual testing methods instead. This would not scale with the many code updates on the app pushed in by the developers during crunch time for release. The QA team simply cannot keep up with the combination of testing many updates with many different test datasets. Thus, some of the test cases must be automated to relieve some execution load from the QA team.

Solution:

1. Simulate User Interactions: To simulate a user interacting with the app using the mouse on the Canvas, Playwright can be used. Among the many features it has, like implicit Actionability Checks, the most useful feature is Tracing. Tracing allows the QA team to perform first level debugging after a failure before submitting a defect ticket with detailed reproduction steps and logs from the trace.

2. Visual Testing Techniques: Due to the rendering variability from each test dataset, Visual Testing on the Canvas can be achieved with the following checks:

• Simple Loading: The app indeed rendered something in the Canvas. 1. The OpenCV’s Contours function can be used. If the bounding box of the largest Contour covers an expected percentage of the Canvas, then the test is considered passed.
• Regions of Interest (ROIs): Inspects certain regions in the Canvas to verify visual correctness. 1. The Scale Invariant Feature Transform (SIFT) feature matching, also found in OpenCV, can inspect the ROIs for visual correctness. As long as the query feature can be found on Canvas, the test is considered passed.
• Rendering Integrity: The rendering in the Canvas is visually similar with the rendering on a different, or pre-updated, rendering engine. 1. The Transformers Model Inferencing together with Perceptual Hashing can be used to determine if there is any degradation in the rendering integrity.

These automated Visual Testing techniques for Canvas can take over some of the more mundane test cases and free-up the QA team for more impactful human-only testing.