230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers

Abdulrahman Alshammari, Paul Ammann, Michael Hilton, Jonathan BellICST 2024 · OpenReview

TL;DR

This paper tackles the challenge of distinguishing flaky test failures from genuine bugs. By analyzing a massive dataset of over 230,000 test failures, the authors empirically evaluate failure de-duplication techniques, finding that their effectiveness varies significantly across projects. The study provides crucial insights into why certain approaches work better in specific contexts, guiding developers on how to best leverage these methods.

What Problem Does This Paper Address?

Automated tests are crucial for software quality, but "flaky" tests pose a significant problem. These tests can unpredictably pass or fail even when no code changes have occurred. While flaky tests can be annoying false alarms, ignoring them entirely can lead to missing actual bugs. The current industry standard for identifying flaky tests is to re-run them: if a test fails and then passes, it's likely flaky. However, this method is inefficient and can miss persistently flaky tests. A more promising approach is "failure de-duplication," where new failures are compared against a history of known flaky and true failures. The core challenge is that flaky failures can mimic real bugs, leading to misclassification and potentially allowing actual defects to slip through. This paper investigates how effectively failure de-duplication can distinguish between flaky and true failures.

Methods

The authors evaluate three primary approaches for failure de-duplication:

  • Text-Based Matching: This method compares the text of failure messages and stack traces to identify similar failures. It focuses on the stack trace lines and exception types, as failure messages can sometimes contain dynamic information (like IP addresses) that makes them unique.
  • Failure Log Classifier: This approach uses machine learning (specifically, decision trees and Naïve Bayes) trained on features extracted from failure logs (like exception type, test name presence in stack traces, etc.) to predict whether a new failure is flaky or true.
  • TF-IDF (Term Frequency-Inverse Document Frequency): This is a standard information retrieval technique used to assess the importance of words in a document. Here, it's applied to failure logs to group similar failures.

The study also involves a novel methodology for creating a dataset of true failures using mutation testing, which is a technique for generating program variants (mutants) to test the effectiveness of the test suite.

Datasets

The core dataset for this study is a newly constructed one comprising 230,439 failure messages from 22 open-source Java projects. This dataset includes:

  • Flaky Failures: Sourced from the "FlakeFlagger" dataset, which contains logs from 498 flaky tests.
  • True Failures: Generated using mutation testing with the PIT tool. This approach aims to create a robust dataset of deterministic failures that are not due to flakiness.

The authors note that while the FlakeFlagger dataset originally covered 24 projects, they could only successfully generate true failure data for 22 of them due to challenges with the mutation testing tool.

Key Insights and Ideas

  • Flaky Failures are Often Repetitive: A significant finding is that flaky test failures tend to be highly repetitive. This means that if a test fails due to flakiness, it's likely to exhibit similar failure symptoms repeatedly, making failure de-duplication a potentially powerful tool.
  • Context Matters Greatly: The effectiveness of failure de-duplication techniques varies dramatically across different projects. What works exceptionally well for one project might be entirely ineffective for another.
  • Exception Type Alone is Insufficient: While certain exception types (like RuntimeException or AssertionError) are more prone to flakiness, relying solely on exception type for classification is often not enough. The specific context within the stack trace is crucial for accurate differentiation.
  • Failure Log Richness is Key: Projects with more specific and informative failure logs (beyond generic assertion errors) tend to see better performance from de-duplication approaches.

Key Results

  • Text-based matching: For some projects, this approach achieved very high precision (up to 99%) and specificity (up to 100%), meaning it was excellent at correctly identifying true failures and not misclassifying them as flaky. However, for other projects, precision dropped as low as 0%, indicating a significant failure to identify true failures.
  • Failure Log Classifier (Decision Tree): This machine learning approach generally performed well, achieving high F1 scores (a combined measure of precision and recall) in many projects.
  • TF-IDF: This method consistently performed well, often outperforming the Failure Log Classifier in terms of minimizing false positives and negatives. It showed particular promise when dealing with projects where other methods struggled.
  • Repetitive Failures: Across all projects, out of 80,530 flaky failures, only 49 were not repetitive when considering all tests within a project. This reinforces the idea that flaky failures often leave a recognizable trace.

Limitations and Future Work

  • Dataset Limitations: The study acknowledges that the true failures were generated via mutation testing, which serves as a proxy for real faults. While mutants are generally considered good substitutes, it's not a perfect representation of all real-world bugs. Additionally, the dataset was collected from a single code revision, and real-world CI environments involve multiple revisions and environments, which could influence flaky failure behavior.
  • Project Variability: The significant variation in performance across projects highlights the difficulty of creating a one-size-fits-all solution. The authors suggest that future work could focus on developing more project-specific or context-aware approaches.
  • Feature Engineering: For the Failure Log Classifier, the authors believe that incorporating more dynamic features or project-specific features could further enhance its performance.
  • Generalization: While the study provides a broad evaluation, extrapolating the "average" performance across all projects is challenging due to the observed variability.

The authors emphasize the need for further research into more advanced test failure clustering and the development of instrumentation-based approaches that can provide richer context for flaky failures, especially for those with generic exception types. They also make their dataset and analysis tools publicly available to encourage further research.

Summary updated

Abstract

Flaky tests are tests that can non-deterministically pass or fail, even in the absence of code changes. Despite being a source of false alarms, flaky tests often remain in test suites once they are detected, as they also may be relied upon to detect true failures. Hence, a key open problem in flaky test research is: How to quickly determine if a test failed due to flakiness, or if it detected a bug? The state-of-the-practice is for developers to re-run failing tests: if a test fails and then passes, it is flaky by definition; if the test persistently fails, it is likely a true failure. However, this approach can be both ineffective and inefficient. An alternate approach that developers may already use for triaging test failures is failure de-duplication, which matches newly discovered test failures to previously witnessed flaky and true failures. However, because flaky test failure symptoms might resemble those of true failures, there is a risk of miss classifying a true test failure as a flaky failure to be ignored. Using a dataset of 498 flaky tests from 22 open-source Java projects, we collect a large dataset of 230,439 failure messages (both flaky and not), allowing us to empirically investigate the efficacy of failure de-duplication. We find that for some projects, this approach is extremely effective (with 100% specificity), while for other projects, the approach is entirely ineffective. By analyzing the characteristics of these flaky and non-flaky failures, we provide useful guidance on how developers should rely on this approach.
BibTeX
@inproceedings{alshammari2024test,
  author = {Abdulrahman Alshammari and Paul Ammann and Michael Hilton and Jonathan Bell},
  title = {{230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers}},
  booktitle = {ICST},
  year = {2024},
  url = {https://openreview.net/forum?id=hoZxdbYxMw},
}
Figure