TL;DR
This paper introduces FIXEVAL, a new benchmark for evaluating automatically generated program fixes, which includes unit tests and considers execution constraints like time and memory. They demonstrate that traditional n-gram-based metrics are insufficient for assessing fix correctness, highlighting the necessity of execution-based evaluation for real-world applicability.
What Problem Does This Paper Address?
Software bugs are a major cost and time sink in development. While automated program repair aims to alleviate this, effectively evaluating the generated fixes is a significant challenge. Existing evaluation methods often rely on metrics like BLEU or edit similarity, which can be misleading. These metrics might deem a syntactically different but functionally correct fix as incorrect, or vice-versa. Furthermore, they don't account for crucial real-world constraints like time and memory limits, which are paramount in competitive programming and often indicative of production code quality. The authors aim to bridge this gap by providing a robust dataset and evaluation framework that better reflects actual program correctness.
Methods
The core of this work is the creation of the FIXEVAL dataset. They leverage submissions from competitive programming platforms (AtCoder and Aizu Online Judge) to create parallel examples of buggy code (submissions that failed tests) and their corresponding correct versions (accepted submissions). Crucially, they pair these with publicly available unit tests designed by domain experts.
For evaluation, they compare traditional n-gram-based metrics (Exact Match, BLEU, Syntax Match, Dataflow Match, CodeBLEU, Compilation Accuracy) against execution-based metrics. The execution-based metrics, Pass@k and Test Case Average (TCA@k), directly run the generated fixes against the collected unit tests, considering time and memory limits. They also experiment with two popular Transformer-based language models (PLBART and CodeT5) as baselines, finetuning them on their dataset.
Datasets
The primary contribution is the FIXEVAL dataset. This dataset is constructed from:
- CodeNet: A large collection of programming problem submissions from AtCoder and Aizu Online Judge.
- AtCoder Unit Tests: Publicly available test cases for competitive programming problems.
FIXEVAL consists of parallel examples of buggy and correct Java and Python programs, along with their associated unit tests. It's significantly larger than existing benchmarks like QuixBugs and Defects4J, offering more lines of code and a wider variety of problems and submission difficulties.
Key Insights and Ideas
- N-gram metrics are insufficient: The paper strongly argues and demonstrates that metrics like BLEU and Exact Match are poor indicators of functional correctness for program fixes. They can be easily fooled by syntactically similar but functionally incorrect code, or penalize valid fixes that differ in structure.
- Execution is king: Direct execution of generated fixes against comprehensive unit tests is the most reliable way to assess correctness. This approach naturally incorporates time and memory constraints, which are vital for real-world code.
- Competitive programming as a proxy: While not identical to professional software development, competitive programming problems offer a rich source of diverse, well-defined tasks with associated test cases and strict performance constraints, making them a valuable proxy for evaluating program repair.
- Verdict information helps: Providing the type of error (e.g., compilation error, wrong answer) as input to the model can significantly improve its ability to generate correct fixes, especially for syntactical issues.
Key Results
- CodeT5 outperforms PLBART: On n-gram-based metrics, CodeT5 generally performed better than PLBART, likely due to its identifier-aware pretraining.
- Execution-based metrics reveal limitations: Naive Copy (simply copying the buggy code) performed surprisingly well on n-gram metrics due to code overlap, but poorly on execution-based metrics, highlighting the inadequacy of n-gram approaches.
- Pass@k and TCA@k are better indicators: The paper shows that execution-based metrics like Pass@k and TCA@k are more reliable for evaluating functional correctness. For instance, CodeT5 achieved significantly higher TCA@k scores (e.g., 32.90 for Java with verdict) compared to its n-gram scores.
- Task difficulty matters: Model performance degrades as the difficulty of the programming problem increases, as expected.
- Verdict type impacts fixability: Compilation errors (CE) are the easiest to fix, while runtime errors (RE) and time limit exceeded (TLE) errors are harder, often requiring more substantial logical changes.
Limitations and Future Work
- Limitations: The authors acknowledge that competitive programming problems, while useful, are not a perfect replica of real-world professional software development. They also note that while the test cases are language-independent, their current dataset is limited to Java and Python.
- Future Work: The authors suggest extending FIXEVAL to other programming languages. They also encourage future research to explore feedback-based approaches (e.g., using an oracle) to further improve bug-fixing models and to develop benchmarks that more closely simulate real-world bug fixing scenarios. They also propose investigating verdict prediction and chain edit suggestion tasks.
Summary updated