FixEval: Execution-based Evaluation of Program Fixes for Programming Problems

Md. Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, Chris Brown

TL;DR

This paper introduces FIXEVAL, a new benchmark for evaluating automatically generated program fixes, which includes unit tests and considers execution constraints like time and memory. They demonstrate that traditional n-gram-based metrics are insufficient for assessing fix correctness, highlighting the necessity of execution-based evaluation for real-world applicability.

What Problem Does This Paper Address?

Software bugs are a major cost and time sink in development. While automated program repair aims to alleviate this, effectively evaluating the generated fixes is a significant challenge. Existing evaluation methods often rely on metrics like BLEU or edit similarity, which can be misleading. These metrics might deem a syntactically different but functionally correct fix as incorrect, or vice-versa. Furthermore, they don't account for crucial real-world constraints like time and memory limits, which are paramount in competitive programming and often indicative of production code quality. The authors aim to bridge this gap by providing a robust dataset and evaluation framework that better reflects actual program correctness.

Methods

The core of this work is the creation of the FIXEVAL dataset. They leverage submissions from competitive programming platforms (AtCoder and Aizu Online Judge) to create parallel examples of buggy code (submissions that failed tests) and their corresponding correct versions (accepted submissions). Crucially, they pair these with publicly available unit tests designed by domain experts.

For evaluation, they compare traditional n-gram-based metrics (Exact Match, BLEU, Syntax Match, Dataflow Match, CodeBLEU, Compilation Accuracy) against execution-based metrics. The execution-based metrics, Pass@k and Test Case Average (TCA@k), directly run the generated fixes against the collected unit tests, considering time and memory limits. They also experiment with two popular Transformer-based language models (PLBART and CodeT5) as baselines, finetuning them on their dataset.

Datasets

The primary contribution is the FIXEVAL dataset. This dataset is constructed from:

  • CodeNet: A large collection of programming problem submissions from AtCoder and Aizu Online Judge.
  • AtCoder Unit Tests: Publicly available test cases for competitive programming problems.

FIXEVAL consists of parallel examples of buggy and correct Java and Python programs, along with their associated unit tests. It's significantly larger than existing benchmarks like QuixBugs and Defects4J, offering more lines of code and a wider variety of problems and submission difficulties.

Key Insights and Ideas

  • N-gram metrics are insufficient: The paper strongly argues and demonstrates that metrics like BLEU and Exact Match are poor indicators of functional correctness for program fixes. They can be easily fooled by syntactically similar but functionally incorrect code, or penalize valid fixes that differ in structure.
  • Execution is king: Direct execution of generated fixes against comprehensive unit tests is the most reliable way to assess correctness. This approach naturally incorporates time and memory constraints, which are vital for real-world code.
  • Competitive programming as a proxy: While not identical to professional software development, competitive programming problems offer a rich source of diverse, well-defined tasks with associated test cases and strict performance constraints, making them a valuable proxy for evaluating program repair.
  • Verdict information helps: Providing the type of error (e.g., compilation error, wrong answer) as input to the model can significantly improve its ability to generate correct fixes, especially for syntactical issues.

Key Results

  • CodeT5 outperforms PLBART: On n-gram-based metrics, CodeT5 generally performed better than PLBART, likely due to its identifier-aware pretraining.
  • Execution-based metrics reveal limitations: Naive Copy (simply copying the buggy code) performed surprisingly well on n-gram metrics due to code overlap, but poorly on execution-based metrics, highlighting the inadequacy of n-gram approaches.
  • Pass@k and TCA@k are better indicators: The paper shows that execution-based metrics like Pass@k and TCA@k are more reliable for evaluating functional correctness. For instance, CodeT5 achieved significantly higher TCA@k scores (e.g., 32.90 for Java with verdict) compared to its n-gram scores.
  • Task difficulty matters: Model performance degrades as the difficulty of the programming problem increases, as expected.
  • Verdict type impacts fixability: Compilation errors (CE) are the easiest to fix, while runtime errors (RE) and time limit exceeded (TLE) errors are harder, often requiring more substantial logical changes.

Limitations and Future Work

  • Limitations: The authors acknowledge that competitive programming problems, while useful, are not a perfect replica of real-world professional software development. They also note that while the test cases are language-independent, their current dataset is limited to Java and Python.
  • Future Work: The authors suggest extending FIXEVAL to other programming languages. They also encourage future research to explore feedback-based approaches (e.g., using an oracle) to further improve bug-fixing models and to develop benchmarks that more closely simulate real-world bug fixing scenarios. They also propose investigating verdict prediction and chain edit suggestion tasks.

Summary updated

Abstract

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FIXEvAL, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixeValoffers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixeValprovides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced.<sup>1</sup><sup>1</sup>https://github.com/FixEval/FixEval_official
BibTeX
@inproceedings{haque2023fixeval,
  author = {Md. Mahim Anjum Haque and Wasi Uddin Ahmad and Ismini Lourentzou and Chris Brown},
  title = {{FixEval: Execution-based Evaluation of Program Fixes for Programming Problems}},
  booktitle = {APR@ICSE},
  year = {2023},
  url = {https://openreview.net/forum?id=chsj0h5WkW},
}
Figure