iFixR: bug report driven program repair

Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Martin Monperrus, Jacques Klein, Yves Le Traon

TL;DR

iFixR is a new automated program repair (APR) pipeline that swaps out the traditional requirement for failing test cases with natural language bug reports. By using information retrieval to locate faults and regression testing to validate patches, it successfully generates correct fixes for real-world bugs in scenarios where traditional tools would be "blind."

What Problem Does This Paper Address?

Most Automated Program Repair (APR) tools follow a "generate-and-validate" loop: they use a failing test case to find the bug (Fault Localization) and then use a suite of passing tests to ensure the proposed fix doesn't break anything else (Validation).

However, there is a massive gap between research and reality. In practice, when a user reports a bug, a reproducing test case often doesn't exist yet. The authors analyzed the popular Defects4J benchmark and found a startling reality: for 92% of the bugs, if you remove the "future" test cases (those written by developers specifically to fix the bug), traditional tools have no failing test to work with. This makes most current APR research "cheating" by using information that wouldn't be available to a developer at the moment a bug is reported.

Methods

iFixR introduces a workflow designed to mimic how a human developer actually works:

  1. IR-based Fault Localization (IRFL): Instead of running tests to see which lines of code are executed (Spectrum-Based Fault Localization), iFixR treats the bug report as a search query. It uses Information Retrieval (IR) techniques to find code statements that are semantically similar to the text in the bug report.
  2. Patch Generation: It uses a library of "fix patterns" (templates derived from common human bug fixes) to mutate the suspicious code.
  3. Regression-only Validation: Since iFixR assumes no failing test case is available, it validates patches by running them against the existing test suite. If a patch passes all current tests (ensuring no regressions), it is considered a candidate.
  4. Prioritization Heuristics: To help developers find the needle in the haystack, iFixR ranks candidate patches using three heuristics:
  5. Minimal Changes: Favoring smaller AST (Abstract Syntax Tree) edits.
  6. Suspiciousness: Favoring locations that had a higher IR similarity score.
  7. Affected Code Elements: De-prioritizing "noisy" mutations, like changing literal values or adding empty method calls.

Datasets

The authors evaluated iFixR on the Defects4J dataset (specifically the Lang and Math projects).

  • The "Cleaned" Benchmark: A key contribution of the paper is a refined version of Defects4J where bugs are explicitly linked to their original bug reports, and "future" test cases are separated from "past" regression tests to create a realistic evaluation environment.

Key Insights and Ideas

  • The Overfitting Irony: Traditional tools that rely on a single failing test case often "overfit"—they find a "patch" that makes that one test pass but doesn't actually fix the underlying logic. The authors found that IR-based localization actually leads to less overfitting because it anchors the fix to the semantic description of the bug rather than just a specific execution path.
  • IRFL is Competitive: Fine-grained, statement-level IR localization proved to be just as accurate as spectrum-based methods, even without the benefit of execution data.
  • The Time-Saving Potential: On average, iFixR could have localized bugs 26 days before the first reproducing test case was ever submitted to the project repositories.

Key Results

  • Repair Success: Using its IR-based localizer, iFixR generated genuine or plausible patches for 21 out of 44 Defects4J faults.
  • Ranking Accuracy: For the faults it could fix, iFixR placed a correct or plausible patch in the top-5 recommendations for 8 out of 13 cases.
  • Comparison: When compared to state-of-the-art APR tools (which usually require failing tests), iFixR's performance was comparable, despite operating under much stricter, more realistic constraints.

Limitations and Future Work

While iFixR is a step toward "test-less" repair, it still has hurdles:

  • The "Donor Code" Problem: Like many APR tools, it relies on finding "ingredients" for fixes within the existing codebase. If the fix requires entirely new logic, iFixR can't generate it.
  • Bug Report Quality: IRFL is only as good as the bug report. If a report is vague or lacks technical keywords, the tool may fail to find the buggy line.
  • Future Directions: The authors suggest integrating iFixR with automated test generation tools. If iFixR could generate a patch and a new test case that matches the bug report's description, it would provide a much stronger "oracle" for correctness.

Summary updated

Abstract

Issue tracking systems are commonly used in modern software development for collecting feedback from users and developers. An ultimate automation target of software maintenance is then the systematization of patch generation for user-reported bugs. Although this ambition is aligned with the momentum of automated program repair, the literature has, so far, mostly focused on generate-and- validate setups where fault localization and patch generation are driven by a well-defined test suite. On the one hand, however, the common (yet strong) assumption on the existence of relevant test cases does not hold in practice for most development settings: many bugs are reported without the available test suite being able to reveal them. On the other hand, for many projects, the number of bug reports generally outstrips the resources available to triage them. Towards increasing the adoption of patch generation tools by practitioners, we investigate a new repair pipeline, iFixR, driven by bug reports: (1) bug reports are fed to an IR-based fault localizer; (2) patches are generated from fix patterns and validated via regression testing; (3) a prioritized list of generated patches is proposed to developers. We evaluate iFixR on the Defects4J dataset, which we enriched (i.e., faults are linked to bug reports) and carefully-reorganized (i.e., the timeline of test-cases is naturally split). iFixR generates genuine/plausible patches for 21/44 Defects4J faults with its IR-based fault localizer. iFixR accurately places a genuine/plausible patch among its top-5 recommendation for 8/13 of these faults (without using future test cases in generation-and-validation).
BibTeX
@inproceedings{koyuncu2019ifixr,
  author = {Anil Koyuncu and Kui Liu and Tegawendé F. Bissyandé and Dongsun Kim and Martin Monperrus and Jacques Klein and Yves Le Traon},
  title = {{iFixR: bug report driven program repair}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2019},
  url = {https://openreview.net/forum?id=HbNV2Z62Ql5},
}

Loading related content...

Figure