FixEval: Execution-based Evaluation of Program Fixes for Competitive Programming Problems

Md. Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, Chris Brown

TL;DR

This paper introduces FIXEVAL, a new benchmark for evaluating automatic program repair techniques. It highlights that traditional n-gram-based metrics are insufficient for assessing the functional correctness of generated code fixes, advocating for execution-based evaluation using comprehensive unit tests.

What Problem Does This Paper Address?

Software bugs are a major cost and time sink in development. While automated program repair aims to alleviate this, effectively evaluating the quality of automatically generated fixes is a significant challenge. Existing benchmarks often lack comprehensive test suites or focus only on superficial code similarity (like n-gram overlap), failing to capture whether a fix actually makes the program functionally correct. This gap makes it hard to reliably compare and improve program repair models.

Methods

The core of this work is the creation of the FIXEVAL dataset. They leverage submissions from competitive programming platforms (AtCoder, Aizu Online Judge) to create pairs of buggy (unaccepted) and correct (accepted) code submissions for the same problem. Crucially, they augment these pairs with the original unit tests provided by these platforms. This allows for execution-based evaluation, where generated fixes are run against these tests to determine functional correctness. They also consider time and memory limits, mimicking real-world competitive programming constraints.

Datasets

  • FIXEVAL Dataset: A newly introduced benchmark comprising buggy and correct program pairs from competitive programming problems. It includes solutions in Java and Python, with an extensive collection of unit tests for evaluation. The dataset is structured to facilitate training and evaluation of program repair models.
  • Existing Benchmarks (for comparison): DeepFix, Review4Repair, Bug2Fix, and GitHub-Python dataset are discussed and compared against FIXEVAL, highlighting their limitations (e.g., lack of test cases, function-level focus, limited error types).

Key Insights and Ideas

  • N-gram metrics are misleading: Traditional metrics like BLEU and Exact Match don't accurately reflect whether a program fix is functionally correct. A fix can have high n-gram overlap but still fail test cases.
  • Execution is king: Evaluating fixes by running them against a comprehensive suite of unit tests is essential for determining true functional correctness.
  • Competitive programming as a proxy: While not identical to professional software development, competitive programming problems offer a rich source of diverse bugs and well-defined test cases, making them suitable for evaluating program repair.
  • Verdict information matters: Providing models with information about the type of error (e.g., Wrong Answer, Compilation Error) can help them pinpoint the bug location and generate more accurate fixes.

Key Results

  • CodeT5 outperforms PLBART: When fine-tuned on FIXEVAL, CodeT5 generally shows better performance across both n-gram-based and execution-based metrics compared to PLBART.
  • Execution-based metrics are superior: Table IV shows that execution-based metrics like Pass@k and TCA@k provide a more realistic assessment of model performance than n-gram metrics (Table III). For instance, Naive Copy (which simply copies the buggy code) performs well on n-gram metrics but poorly on execution-based ones.
  • Task difficulty impacts performance: As problem difficulty increases (from A to E), model performance (measured by Pass@k) degrades, indicating that fixing bugs in more complex problems is harder.
  • Verdict type influences fixability: Compilation errors (CE) are generally easier to fix than runtime errors (RE) or time limit exceeded (TLE) errors, which often require more significant logical changes.

Limitations and Future Work

  • Limitations: The authors acknowledge that competitive programming problems, while useful, are not a perfect replica of real-world software development bugs. They also note that the unit tests, while provided by domain experts, were not manually validated by the authors.
  • Future Work: The paper suggests extending FIXEVAL to other programming languages. They also encourage future research to explore feedback-based approaches (e.g., using an oracle) for improving bug-fixing models and to develop benchmarks that more closely simulate real-world software bugs.

Summary updated

Abstract

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixEval provides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced at https://github.com/mahimanzum/FixEval.
BibTeX
@inproceedings{haque2023fixeval,
  author = {Md. Mahim Anjum Haque and Wasi Uddin Ahmad and Ismini Lourentzou and Chris Brown},
  title = {{FixEval: Execution-based Evaluation of Program Fixes for Competitive Programming Problems}},
  booktitle = {APR@ICSE},
  year = {2023},
  url = {https://openreview.net/forum?id=chsj0h5WkW},
}
Figure