Mitigating Flaky Tests in Automated Software Testing

28 concepts20 papers1 proposal

Automated testing relies on a fundamental assumption: determinism. If the underlying source code has not changed, the test outcome should remain identical across executions. However, in modern continuous integration (CI) pipelines, this assumption frequently breaks down due to flaky tests—test cases that intermittently pass and fail under identical code conditions.

As highlighted by Nikhil Singh in CI/CD Pipeline for Web Applications, these false alarms degrade the reliability of the automated feedback loop, inducing developer alert fatigue that can lead to critical system failures being ignored. The scale of this problem is massive; for instance, Artur Andrzejak et al. observed in Testing Very Large Database Management Systems: The Case of SAP HANA that flakiness can affect up to 20% of a large-scale test suite, severely draining engineering productivity and wasting computational resources. Even in experimental settings, as noted by Ramayanam Venkata et al. in The Effect of Replicated Archetypes on Machine Learning, systemic software instability can completely undermine the evaluation of underlying methodologies.

The Hidden Cost: Hiding Real Bugs and Breaking Tools

The intuitive reaction to an intermittent test failure in a CI pipeline is often to simply rerun the test. If it passes on a subsequent try, developers frequently assume the failure was "just flakiness" and proceed. However, Guillaume Haben et al. demonstrate in A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests that this practice is highly dangerous. By treating intermittent failures as mere test artifacts, teams risk masking genuine, non-deterministic bugs in the production code itself, effectively shipping unstable software to production. This non-determinism also complicates automated patch validation in complex environments like operating system kernels, where crashes may not trigger consistently during evaluation cycles (Kareem Shehada et al. in Rethinking Kernel Program Repair: Benchmarking and Enhancing LLMs with RGym).

Furthermore, flakiness breaks downstream automated tools that assume perfect determinism:

Exposing Flakiness Through Resource Manipulation

How do we identify a flaky test before it pollutes the CI pipeline? The traditional approach is brute-force repetition: executing a suspicious test case multiple times (e.g., 20 consecutive runs) under identical conditions to see if the outcome changes (Yu Pei et al. in An empirical study of async wait flakiness in front-end testing). However, as Abdulrahman Alshammari et al. point out in 230, 439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers, simple rerunning is highly inefficient and often fails to uncover persistently flaky behavior.

A more elegant, proactive mechanism leverages the relationship between flakiness and hardware constraints. Denini Silva et al. introduce the concept of Resource-Affected Flaky Tests (RAFTs) in The Effects of Computational Resources on Flaky Tests. The intuition is simple: many flaky tests fail because of subtle timing discrepancies or race conditions. By intentionally restricting computational resources (such as CPU or memory), developers can amplify these timing gaps, forcing flaky tests to fail much more frequently and making them easier to catch cost-effectively.

Root Cause Localization and Classification

Once a test is known to be flaky, developers need to know where the flakiness originates and why it happens.

To find the "where," Guillaume Haben et al. in What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness repurposed Spectrum-Based Fault Localization (SBFL). By combining test execution histories with change metrics, their approach pinpoints the specific source code classes responsible for the non-deterministic behavior, drastically reducing manual debugging overhead.

To understand the "why," the authors of FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning (Amal Akli et al.) utilized few-shot learning and CodeBERT to automatically categorize flaky tests by their underlying root causes, such as "Async waits" or "Concurrency." Knowing the category allows developers to apply targeted fixes rather than guessing blindly.

Moving Beyond "Sleep" Commands: Effective Mitigation

The most common anti-pattern in fixing flaky tests—especially in UI and front-end testing—is adding arbitrary sleep or timeout commands to wait for an application state to update. Both Yu Pei et al. and Alan Jesus Romano et al. (in An Empirical Analysis of UI-based Flaky Tests) demonstrate that this approach is fundamentally flawed. Flakiness is a symptom of how a test observes and interacts with application state; when resource availability shifts, arbitrary timers inevitably fail. Instead, the authors show that effective mitigation requires refactoring the test logic to use explicit synchronization points (e.g., polling for a specific DOM element to appear).

For complex, stateful tests, Marcel Böhme suggests in Assurance in software testing: a roadmap that advanced virtualization techniques can be deployed to isolate and "unflake" tests by distinguishing between true non-determinism and lingering state pollution. Finally, when dealing with historical test logs, failure de-duplication techniques can group similar test failures together, helping separate transient flaky noise from genuine regressions (Abdulrahman Alshammari et al.).

Key Foundational Work

Assurance in software testing: a roadmap (2019) — contributed 2 concept elements to this direction.
An Efficient Algorithm to Identify Minimal Failure-Causing Schemas from Exhaustive Test Suite (2016) — contributed 1 concept elements to this direction.
The Effect of Replicated Archetypes on Machine Learning (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_flaky_tests_b5c235b9,
  title = {flaky tests},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/flaky-tests/rdr_b5c235b9f0bb45f98a73bcbec720887d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_flaky_tests_b5c235b9_v1,
  title = {flaky tests},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/flaky-tests/rdr_b5c235b9f0bb45f98a73bcbec720887d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake