Attention Please: Consider Mockito when Evaluating Newly Proposed Automated Program Repair Techniques

Shangwen Wang, Ming Wen, Xiaoguang Mao, Deheng Yang

TL;DR

Automated Program Repair (APR) techniques are evaluated on benchmarks like Defects4J, but the Mockito project's bugs have been largely ignored. This paper argues that this oversight leads to an incomplete understanding of APR tool performance, potentially due to overfitting. The authors demonstrate that current state-of-the-art APR tools perform poorly on Mockito bugs, even when the bugs aren't inherently more complex, and identify a novel repair action ("external package addition") that might explain this deficiency.

What Problem Does This Paper Address?

The field of Automated Program Repair (APR) aims to automatically fix bugs in software, reducing the significant cost and effort of manual debugging. To assess the effectiveness of new APR techniques, researchers rely on standardized benchmarks. Defects4J is a widely adopted benchmark for Java programs, containing real-world bugs extracted from version control histories. However, when Defects4J was updated to include bugs from the Mockito project, many subsequent APR studies overlooked these new additions. This raises a critical question: does ignoring the Mockito bugs lead to a biased evaluation of APR techniques? The authors hypothesize that current APR tools might be "overfitting" to the older, more established projects within Defects4J, and that their performance on Mockito bugs could reveal limitations not apparent in other evaluations.

Methods

The paper employs an empirical approach to investigate the performance of APR techniques on Mockito bugs. Their study is structured into four phases:

  1. Patch Characteristics Comparison: They analyze the complexity of patches for Mockito bugs by comparing metrics like patch size, number of chunks, modified files, and modified methods against non-Mockito bugs in Defects4J.
  2. Repair Pattern Analysis: They examine whether Mockito bugs require unique repair patterns not found in other projects, and if existing APR techniques can handle bugs with similar repair patterns from other projects.
  3. APR Tool Evaluation: They conduct experiments with two state-of-the-art search-based APR tools (SimFix and CapGen) on Mockito bugs. They also incorporate results from a previous study evaluating a semantic-based tool (Nopol) on Mockito.
  4. Unique Repair Action Identification: They perform a fine-grained analysis of Mockito bug fixes to identify any novel repair actions that might be overlooked by current APR techniques.

Datasets

The primary dataset used is Defects4J, specifically version 1.1.0, which includes 38 bugs from the Mockito project alongside bugs from other projects like JFreechart, Closure compiler, Apache commons-lang, Apache commons-math, and Joda-Time. The authors do not introduce a new benchmark but rather highlight the underutilization of an existing component (Mockito) within a well-established benchmark.

Key Insights and Ideas

  • Overfitting to the Familiar: The core idea is that APR tools might be implicitly optimized for the types of bugs and fixes found in older projects within Defects4J, leading to poor performance on newer additions like Mockito.
  • Complexity is Not the Issue: Mockito bugs are not inherently more complex in terms of patch characteristics (size, number of modified files/methods) than bugs from other Defects4J projects. This suggests the problem lies elsewhere.
  • Existing Patterns are Insufficient: While Mockito bugs generally follow established repair patterns, current APR tools struggle to apply them effectively to Mockito code.
  • "External Package Addition" is a Novel Repair Action: A significant finding is the identification of a unique repair action in Mockito bugs: the need to import external packages that are not part of the original project but are present in the test code. This action is crucial for fixing a substantial portion of Mockito bugs and is largely ignored by current APR techniques.

Key Results

  • Poor APR Performance on Mockito:
    • SimFix failed to fix any Mockito bugs, with many resulting in timeouts or failures to find a valid patch.
  • CapGen also failed to fix any Mockito bugs, generating many plausible patches but none that were correct.
  • Nopol, a semantic-based tool, only managed to fix one Mockito bug, resulting in a very low repair rate (5% in the paper's analysis, 2.6% after manual verification).
  • Patch Complexity: Statistical tests showed no significant difference in patch complexity (size, chunks, modified files/methods) between Mockito bugs and non-Mockito bugs.
  • Repair Patterns: Mockito bugs utilize repair patterns that are also found in other Defects4J projects, and existing techniques have successfully fixed similar bugs in those projects.
  • Unique Repair Action: "External package addition" was identified as a unique repair action in Mockito bugs, accounting for 28.9% of the bugs in the project. This involves importing necessary code from test files into the source code.

Limitations and Future Work

  • Execution Environment: The experiments for SimFix were conducted on a machine with 2GB RAM, while the original evaluation used 8GB. The authors acknowledge that lower computational resources might affect performance, though they state no specific study confirms this for APR tools.
  • Generalization: The study evaluated only three APR tools (SimFix, CapGen, Nopol). While these are considered representative (state-of-the-art search-based and widely used semantic-based), the findings might not universally apply to all APR techniques.
  • Future Work: The authors suggest that incorporating Mockito bugs into evaluation sets is crucial to avoid bias and gain a more accurate understanding of APR progress. They also propose that the identified "external package addition" repair action, and the method of copying packages from test code to source code, could be integrated into existing APR techniques to improve their effectiveness. Further research could explore evaluating more APR tools on Mockito and investigating the impact of hardware configuration on APR performance.

Summary updated

BibTeX
@inproceedings{wang2019attention,
  author = {Shangwen Wang and Ming Wen and Xiaoguang Mao and Deheng Yang},
  title = {{Attention Please: Consider Mockito when Evaluating Newly Proposed Automated Program Repair Techniques}},
  booktitle = {EASE},
  year = {2019},
  doi = {10.1145/3319008.3319349},
}

Loading related content...

Figure