Fixing Code Generation Errors for Large Language Models

Hao Wen, Yueheng Zhu, Chao Liu, Xiaoxue Ren, Weiwei Du, Meng YanarXiv 2024 · OpenReview

Abstract

Function-level code generation leverages foundation Large Language Models (LLMs) to automatically produce source code with expected functionality. Despite advancements, generation involves many errors. This paper conducts an empirical study on 12,837 code generation errors from 14 representative LLMs, leading to 19 categories of error causes. Based on these findings, we propose LlmFix, a rule-based method that addresses three common error types: indentation, redundant code, and missing modules. Evaluations show LlmFix achieves a 17.1% fix rate on the LlmErrorEval dataset and improves code generation accuracy by an average of 7.5% across 14 LLMs.

Introduction

Large Language Models (LLMs) like CodeLlama and GPT-3.5 have revolutionized software development through tools like GitHub Copilot. However, these models frequently produce code with structural errors—missing imports, hallucinated extra functions, or incorrect indentation—that lead to immediate execution failures. While existing solutions use heavy static analysis or additional "fixing" LLMs, they often overlook the root causes of these common errors. This research investigates these failures through a massive empirical study and introduces LlmFix, a lightweight, rule-based tool designed to correct structural noise and significantly boost generation accuracy with minimal computational overhead.

Empirical Study of Generation Errors

The researchers reproduced 14 representative LLMs (including the CodeLlama, WizardCoder, and Phi families) on the HumanEval benchmark to analyze failure modes. They collected 12,837 failed code snippets and manually categorized them into 19 root causes. A significant finding was the variance in model performance during reproduction compared to reported figures, highlighting the instability of zero-shot code generation.

Accuracy distribution of 14 representative LLMs on the HumanEval dataset, illustrating the performance variance across multiple runs.

Analysis revealed that while 63.6% of errors are logic-based (AssertionErrors), a substantial portion are structural. For instance, "Function Overflow"—where LLMs continue generating code after a function is complete—is a major source of syntax errors.

LlmFix: A Rule-Based Fixing Method

Based on the fixability analysis of the 19 identified causes, the authors developed LlmFix, a three-step pipeline targeting predictable structural errors:

  1. Code Filtering: Standardizes indentation and removes problematic blocks like input() calls that hang test suites.
  2. Code Truncation: Iteratively removes trailing incomplete functions or "rambling" text generated after the primary function.
  3. Importing Missing Modules: Automatically adds necessary standard library imports (e.g., import math) when a NameError is detected.

Experimental Results

LlmFix was evaluated on HumanEval, MBPP, and a new specialized dataset, LlmErrorEval. The tool achieved a 17.1% fix rate on LlmErrorEval, outperforming the state-of-the-art LLM-based debugger (LDB) by 8.9%. When integrated with LDB, the fix rate rose to 24.9%, demonstrating that rule-based fixing complements LLM-based debugging by handling structural "noise."

Detailed performance metrics of the evaluated LLMs, showing the gap between reported accuracy and reproduced mean accuracy.

Crucially, LlmFix is highly efficient, taking an average of just 11.5ms to fix an error, whereas LLM-based debuggers require multiple model inferences taking seconds or minutes. Applying LlmFix as a post-processing step improved the Pass@1 accuracy of the 14 LLMs by an average of 7.5%.

Conclusion

LlmFix demonstrates that many LLM generation errors are structural and can be corrected using simple, deterministic rules. By addressing these "low-hanging fruit" errors, LlmFix provides a highly efficient post-processing step that significantly boosts the reliability of LLM-generated code without the need for expensive additional GPU cycles.

Paper Citation: Hao Wen, Yueheng Zhu, Chao Liu, Xiaoxue Ren, Weiwei Du, and Meng Yan.

  1. Fixing Function-Level Code Generation Errors for Foundation Large Language Models. ACM Trans. Softw. Eng. Methodol. 1, 1, Article 1 (August 2024).

Summary updated

Abstract

Function-level code generation leverages foundation Large Language Models (LLMs) to automatically produce source code with expected functionality. It has been widely investigated and applied in intelligent programming assistants, such as GitHub Copilot, to enhance software development productivity. Despite advancements in foundation LLMs, the generation involves many errors. Existing studies leverage static analysis tools (e.g., TBar) or add another fixing LLM (i.e., LDB) to post-process these errors. However, there are still many errors remaining to be solved because their root causes have not been investigated yet, making it challenging to design better fixing tools. In this paper, we first conducted an empirical study on the generation errors. Specifically, we reproduced 14 representative LLMs on the HumanEval dataset and verified their correctness. We obtained 12,837 code generation errors and conducted an analysis of their causes, leading to 19 categories of error causes. Our empirical analysis indicated that three of these causes can be directly fixed. Based on the findings, we proposed a fixing method called LlmFix, which addresses these three types of errors through a three-step process: filtering code for indentation correction, truncating redundant generated code, and importing missing modules. Evaluations of LlmFix are conducted from two perspectives: its performance on error-fixing tasks and its impact on improving function-level code generation tasks. For error fixing performance, we built an evaluation dataset LlmErrorEval. Experimental results show that LlmFix achieves a fix rate of 17.1% outperforming the best LDB by 8.9%. For code generation improvements, evaluations of LlmFix on both the HumanEval and MBPP datasets demonstrate its effectiveness, improving code generation accuracy by an average of 7.5% across 14 LLMs.
BibTeX
@misc{wen2024fixing,
  author = {Hao Wen and Yueheng Zhu and Chao Liu and Xiaoxue Ren and Weiwei Du and Meng Yan},
  title = {{Fixing Code Generation Errors for Large Language Models}},
  year = {2024},
  eprint = {2409.00676},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.00676},
}
Figure