Managing Non-Determinism in Automated Testing Pipelines

This direction is split into other directions: flaky tests. Its last version stays readable and citable here.

Version 1 of 122 papers29 conceptsfrozen 2026-10-04

Modern software engineering relies on automated feedback loops, yet these loops are increasingly compromised by "flaky" tests—tests that yield inconsistent pass or fail results on identical code. This non-determinism creates a "ghost in the machine" that erodes developer trust, consumes massive computational resources, and masks genuine software regressions.

The Cost of Inconsistency

In large-scale industrial environments, the scale of flakiness is staggering. Artur Andrzejak and Thomas Bach note that in systems like SAP HANA, up to 20% of a test suite can exhibit flaky behavior, necessitating millions of daily executions just to maintain a baseline of quality Testing Very Large Database Management Systems: The Case of SAP HANA. This noise is not merely an operational nuisance; it fundamentally undermines the reliability of Continuous Integration (CI) signals. When a test failure might be a false alarm, developers often resort to "rerun until pass" strategies, which Guillaume Haben and Mike Papadakis warn can lead to shipping unstable code that exhibits the same non-determinism in production A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests.

Discerning Failures from Flakiness

The research direction has shifted from simply identifying flaky tests to distinguishing between a flaky test and a flaky failure. Guillaume Haben demonstrates that labeling a test as generally flaky is dangerous: such tests often retain the ability to detect real bugs, but their signals are ignored once they are "blacklisted" The Importance of Discerning Flaky from Fault-triggering Test Failures. To address this, tools like FAIR use machine learning to analyze test metadata and logs, identifying legitimate failures with high accuracy without the cost of repeated reruns Discerning Legitimate Failures From False Alerts.

Impact on Automated Repair and LLMs

The rise of Large Language Models (LLMs) in software engineering has introduced new dimensions to the flakiness problem. LLMs are now used to generate tests, but Alexander Berndt finds these generated tests are often flakier than human-written ones because they fail to account for non-deterministic system behaviors like SQL result ordering On the Flakiness of LLM-Generated Tests. Furthermore, flakiness acts as a "poison" for automated program repair. Maxime Cordy and Mark Harman show that even tiny amounts of noise can derail repair algorithms, making it impossible to tell if a patch actually fixed a bug or if the test just happened to pass FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment.

Controlled Experimentation and Scaling

To combat this, researchers are developing frameworks like FLAKER to intentionally inject non-determinism into stable suites, creating the large-scale datasets needed to train better detection tools On the Use of Mutation in Injecting Test Order-Dependency. Meanwhile, in the realm of LLM evaluation, Bradley Brown and Quoc V Le suggest that while repeated sampling can scale model performance, the underlying unreliability of test benchmarks remains a critical bottleneck for verifying those solutions Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_managing_non_determinism_in_automated_te_6f7e95c4,
  title = {Managing Non-Determinism in Automated Testing Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/managing-non-determinism-in-automated-testing-pipelines/rdr_6f7e95c4ca144ebba4620f92d0874c2a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_managing_non_determinism_in_automated_te_6f7e95c4_v1,
  title = {Managing Non-Determinism in Automated Testing Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/managing-non-determinism-in-automated-testing-pipelines/rdr_6f7e95c4ca144ebba4620f92d0874c2a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}