Eradicate Flaky Tests

Have you heard of “flaky tests”?

Many articles, blog posts, podcasts, and conference talks about what are “Flaky tests” and how to avoid them.

Some of the ideas proposed are:

* Automatically rerun failed tests a couple of times, and hope they pass
* Automatically retry certain operations in the test (ex: retry click/checking for visibility of elements, etc.) and hope the test can proceed

Unfortunately, I do not agree with the above ideas, and I would term these as anti-patterns for fixing flaky/intermittent tests.

We need to understand the reasons for flaky/intermittent tests. Some of these reasons could be because of issues like:

* timing issues (i.e. page loading taking time)
* network issues
* browser-timing issues (different for different browsers/devices)
* data related (dynamic, changing, validity, etc.)
* poor locator strategy (ex: weird & hard-wired xpaths/locators)
* environment issue
* actual issue in the product-under-test

In this session, with the help of demos, we will look at the following techniques you can use to reduce/eliminate the flakiness of your test execution:

* Reduce the number of UI tests
* Use Visual Assertions instead of Functional Assertions
* Remove external dependencies via Intelligent Virtualization

Demos will be done using the following and a sample code will be shared with the participants

* Specmatic
* teswiz
* Applitools Eyes

This will be done as live a demo. 45-60 min is a good time for this session. Internet connectivity will be required.

You can find links to some of my earlier conference videos here - https://www.youtube.com/user/abagmar/videos, and my slides here - http://slideshare.net/abagmar

Automating the real-user scenarios across multi-apps, and multi-devices

Simulating real-user scenarios as part of your automation is a solved problem. You need to understand the domain, the product, the user, and then define and implement your scenario.

But there are some types of scenarios that are complex to implement. These are the real-world scenarios having multiple personas (users) interacting with each other to use some business functionalities. These personas may be on the same platform or different (web / mobile-web / native apps / desktop applications).

Example scenarios:
- How do you check if more than 1 person is able to join a zoom / teams meeting? And that they can interact with each other?
- How do you check if the end-2-end scenario that involves multiple users, across multiple apps works as expected?
Given user places order on Amazon (app / browser)
When delivery agent delivers the order (using Delivery app)
Then user can see the order status as "Delivered"

Even though we will automate and test each application in such interactions independently, or test each persona scenarios independently, we need a way to build confidence that these multiple personas and applications can work together. These scenarios are critical to automate!

In this session, I will demonstrate teswiz, an open-source framework can easily automate these multi-user, multi-app, multi-device scenarios. I will also mention how to run these tests on local and in CI pipelines.

Example: Multi-user, Multi-device test scenario
@multiuser-android-web @videoRequest
Scenario: Host (on Android) requests Guest (on Web) to start video
Given "Host" signs up (using API), logs-in and starts an instant meeting on "android"
And "Guest" joins the meeting from "web"
When "Host" asks "Guest" to turn on the "video"
Then "Guest" should be able to select "later" from video request
And "Host" should receive a toast saying, "Guest" will switch on the "video" later
When "Host" asks "Guest" to turn on "video"
Then "Guest" should be able to select "allow" from video request

Example: Multi-user, Multi-app, Multi-device test scenario
@multiuser-android
Scenario: Verify my order status when delivery-agent delivers the app successfully
Given "I" order a "One Plus 9" phone from "amazon" using the "android app"
When "delivery-agent" delivers the app using the "delivery-agent" "android app"
Then "I" can see the item delivered in Order Status

Teswiz enables, and guides you to implement your automated tests while adhering to the principles of test automation, like - independent tests that run in parallel, against multiple environments using environment-specific-test-data and generate rich and contextual reports (and test execution trends) in reportportal.

Test coverage is increased by using Applitools Visual AI, along with Applitools Ultrafast Test Cloud.

In addition, teswiz takes away the pain of managing your browsers, and android / ios / windows devices for automation. The automated tests can run on local browsers / devices, or against any cloud provider, such as HeadSpin, BrowserStack, SauceLabs, pCloudy.

The following features makes teswiz unique:

The ability to do:
- Multi-user scenario automation between all platforms (android, iOS, Web, Windows desktop applications)
- Managing browsers / devices and parallel execution automatically
- Completely configurable options - means no code change required to run different combinations of tests, on demand
- Integration with Applitools Visual AI and Applitools Ultrafast Test Cloud
- Rich contextual reports (including screenshots, browser / device logs) and trend analysis via ReportPortal.io

- This will be a live demo-based session.
- I will need Internet access for the same
- Target audience: QA, SDET, developers, leads and managers involved with building and testing applications that have multiple personas interacting with each other - ex: chat, video calls, etc.

Break the Test Data Bottleneck: Real Scenarios on Real Systems—Validated with Visual AI

Modern testing is still stuck behind one stubborn problem: test data.

We spend hours creating it, maintaining it, and coordinating across teams—yet still struggle to test the scenarios that matter most: failures, edge cases, inconsistent responses, and real user journeys across distributed systems.

In API-driven architectures, this problem is amplified. A single user flow depends on multiple services, each with its own failure modes, latency, and evolving contracts. Reproducing these conditions reliably is difficult, and even when we do, validation often breaks down—leaving gaps between what we test and what actually happens in production.

What if we could test real scenarios on real systems—and validate them visually with high confidence—without writing more code?

In this session, we’ll go beyond traditional approaches and show how to combine live traffic simulation using Specmatic Live Proxy with AI-powered visual validation using Applitools Visual AI. This enables teams to simulate real-world conditions—like API failures, latency, and unexpected data—and validate UI outcomes deterministically across these scenarios.

Rather than showcasing multiple disconnected capabilities, we’ll focus on a single critical user flow and evolve it across three targeted scenarios:

- Failure scenario → demonstrates controlled backend condition simulation
- Edge case (empty/inconsistent response) → exposes UI regressions beyond status codes
- Latency scenario → validates resilience of UI behavior and visual checkpoints

We’ll also make explicit what changes in the test configuration between scenarios—so the approach is clear, repeatable, and not “magic.”

You’ll see how this approach:

- Eliminates dependency on complex test data
- Expands coverage to include failure and edge scenarios
- Reduces brittle assertions and test code
- Improves confidence with consistent, AI-driven validation

Through a live demo on a deployed system, we’ll simulate real-world conditions and validate UI behavior across scenarios—without changing backend systems or duplicating test logic.

If your goal is to test what truly happens in production—and validate it with confidence—this session will give you a practical, scalable way forward.

Learning outcome:
- Define clear simulation boundaries by deciding what to intercept vs what must remain real in API-driven flows
- Stabilize UI validation in dynamic applications using visual AI—reducing flakiness and brittle assertions
- Increase scenario coverage without test duplication by designing a single reusable test flow across multiple backend conditions
- Differentiate real regressions from noise by tuning visual validation for meaningful, user-visible changes
- Combine backend simulation with validation effectively to test production-like scenarios with higher confidence

Break the API Test Data Bottleneck: Real Scenarios on Real Systems

Modern testing is still stuck behind one stubborn problem: test data.

We spend hours creating it, maintaining it, waiting for it, resetting it, and coordinating with other teams just to get the “right” scenario. And even after all that effort, we still struggle to test the situations that matter most—failures, edge cases, slow responses, missing data, and real user journeys across connected systems.

What if we stopped chasing perfect test data and started testing with realistic scenarios on real systems instead?

In this session, I’ll show how teams can capture live traffic and selectively simulate specific scenarios using tools like Specmatic Live Proxy, enabling testing on deployed systems without changing backend code or disrupting shared environments. This approach allows teams to validate difficult scenarios like service failures, empty states, unexpected responses, and latency in a controlled and repeatable way.

Through practical examples and a live demo, you’ll see how to move beyond brittle mocks and incomplete test setups, and instead create meaningful tests that reflect how systems actually behave.

If your team is tired of test data delays, environment dependency, and gaps in coverage, this session will give you a practical way to test smarter, faster, and with far more confidence.

Learning outcome:
- Identify when test data is limiting the testing effectiveness and confidently decide when simulation is a better approach.
- Design tests for real-world failure and edge scenarios that are typically skipped due to complexity or environment constraints.
- Leverage live system traffic to create realistic and repeatable test conditions, eliminating the need for complex data setup.
- Apply selective API simulation using tools like Specmatic Live Proxy to control specific scenarios while keeping the rest of the system fully functional.
- Implement a practical, incremental strategy to improve test coverage and confidence—especially for negative and resilience testing—without slowing down delivery.

Architecture & Design: The Secret Weapon for Scaling Test Execution in the Age of AI

Most test automation suites don’t fail because teams write bad code; they fail because the underlying architecture can’t handle scale. When you test a business operation across multiple brands, markets, and vendor APIs, you quickly hit a combinatorial explosion of brittle, duplicated scripts. Bringing AI code generators into that mess only generates bad tests faster.

In this talk, I’ll share how we tackled this at a large multi-brand gaming operator. On paper, testing one operation across 6 integrations, 5 brands, and 3 markets meant 1,080 combinations. Two previous frameworks had already stalled trying to automate it.

Instead of writing more scripts, we treated it as an architecture problem:
1. We modeled the matrix as a tree instead of a cross-product, collapsing 1,080 combinations to 15 real ones.
2. We strictly separated business intent from wire-protocol dialects.
3. We injected coordinates at runtime.

The result: adding a new integration dropped from days of manual work to under an hour, with zero changes to existing tests. I’ll also show why these exact architectural seams are essential to prevent AI drift and keep agentic systems reliable.

Beyond the DOM: Practical Test Automation for Canvas, Games, and Spatial UIs

Standard UI automation tools like Selenium, Playwright, and Appium rely on a clean DOM and accessibility tree to find elements. But what happens when your application renders directly to a `canvas` element or WebGL?

Whether you're testing HTML5 games, interactive maps, casino slots, or Flutter web apps, standard locators simply hit a wall. To your test framework, the entire screen is an opaque black box—no tags, no IDs, and no text nodes. Most teams either give up and test manually, or fall into the trap of brittle pixel-diffing and hardcoded click coordinates that break the moment screen resolutions change.

In this technical session, I’ll share how we solved this inside teswiz, an open-source automation engine. We’ll look at a resilient architecture that combines multi-engine OCR for reading dynamic text, computer vision template matching with confidence thresholds, and viewport-normalized math to handle different screen densities.

You’ll walk away with practical patterns to automate canvas journeys reliably, keeping your tests readable, robust, and completely independent of resolution changes.

Giving AI Agents Hands & Eyes: Autonomous Testing Beyond Chatbots with MCP (Model Context Protocol)

Most discussions about AI in testing are stuck on chatbot demos: copying user stories into ChatGPT, asking for test cases, and pasting the code back into an IDE. That isn’t autonomous testing; it’s just interactive autocomplete.

True agentic testing requires giving AI models real "hands and eyes"—the ability to explore running applications, inspect logs, trigger test suites, and diagnose failures without a human babysitting every step. But building custom integrations to connect LLMs to browsers, devices, and test runners is fragile and hard to maintain.

That’s where the Model Context Protocol (MCP) changes the game. MCP provides an open standard for exposing tools, system state, and context to AI models.

In this session, I’ll share what I’ve learned building teswiz-mcp and integrating agentic workflows into test infrastructure. I’ll demonstrate how to expose test runners and inspection utilities as standardized MCP tools, show an agent diagnosing and triaging a failure across logs and UI states, and cover the practical guardrails you need to keep autonomous agents safe and reliable.

Decoupling at Scale: Solving the Multi-System Test Data Crisis

Ask any engineering team why their test suite is slow or flaky, and you’ll rarely hear them blame test design. The real culprits are almost always fragile shared staging environments and broken test data. When running an end-to-end test requires six backend services and two third-party vendor APIs to all be healthy simultaneously, builds inevitably turn red for reasons that have nothing to do with product bugs.

Handcrafted mocks seem like an easy out, but they quickly drift from reality, giving you green builds right up until things fail in production.

In this talk, I’ll share how to break free from this staging dependency. Drawing on enterprise implementations, I’ll show how combining contract-driven development (using open tools like Specmatic) with smart service virtualization enables fully isolated, hermetic testing.

You’ll see how to replace hours of flaky end-to-end staging runs with 10-second test suites that run reliably offline on a laptop or in parallel CI containers, all while guaranteeing strict adherence to production API contracts.