Standard UI automation tools like Selenium, Playwright, and Appium rely on a clean DOM and accessibility tree to find elements. But what happens when your application renders directly to a `canvas` element or WebGL?

Whether you're testing HTML5 games, interactive maps, casino slots, or Flutter web apps, standard locators simply hit a wall. To your test framework, the entire screen is an opaque black box—no tags, no IDs, and no text nodes. Most teams either give up and test manually, or fall into the trap of brittle pixel-diffing and hardcoded click coordinates that break the moment screen resolutions change.

In this technical session, I’ll share how we solved this inside teswiz, an open-source automation engine. We’ll look at a resilient architecture that combines multi-engine OCR for reading dynamic text, computer vision template matching with confidence thresholds, and viewport-normalized math to handle different screen densities.

You’ll walk away with practical patterns to automate canvas journeys reliably, keeping your tests readable, robust, and completely independent of resolution changes.