Abstract
Function-level code generation leverages foundation Large Language Models (LLMs) to automatically produce source code with expected functionality. Despite advancements, generation involves many errors. This paper conducts an empirical study on 12,837 code generation errors from 14 representative LLMs, leading to 19 categories of error causes. Based on these findings, we propose LlmFix, a rule-based method that addresses three common error types: indentation, redundant code, and missing modules. Evaluations show LlmFix achieves a 17.1% fix rate on the LlmErrorEval dataset and improves code generation accuracy by an average of 7.5% across 14 LLMs.
Introduction
Large Language Models (LLMs) like CodeLlama and GPT-3.5 have revolutionized software development through tools like GitHub Copilot. However, these models frequently produce code with structural errors—missing imports, hallucinated extra functions, or incorrect indentation—that lead to immediate execution failures. While existing solutions use heavy static analysis or additional "fixing" LLMs, they often overlook the root causes of these common errors. This research investigates these failures through a massive empirical study and introduces LlmFix, a lightweight, rule-based tool designed to correct structural noise and significantly boost generation accuracy with minimal computational overhead.
Empirical Study of Generation Errors
The researchers reproduced 14 representative LLMs (including the CodeLlama, WizardCoder, and Phi families) on the HumanEval benchmark to analyze failure modes. They collected 12,837 failed code snippets and manually categorized them into 19 root causes. A significant finding was the variance in model performance during reproduction compared to reported figures, highlighting the instability of zero-shot code generation.
Analysis revealed that while 63.6% of errors are logic-based (AssertionErrors), a substantial portion are structural. For instance, "Function Overflow"—where LLMs continue generating code after a function is complete—is a major source of syntax errors.
LlmFix: A Rule-Based Fixing Method
Based on the fixability analysis of the 19 identified causes, the authors developed LlmFix, a three-step pipeline targeting predictable structural errors:
- Code Filtering: Standardizes indentation and removes problematic blocks like
input()calls that hang test suites. - Code Truncation: Iteratively removes trailing incomplete functions or "rambling" text generated after the primary function.
- Importing Missing Modules: Automatically adds necessary standard library imports (e.g.,
import math) when aNameErroris detected.
Experimental Results
LlmFix was evaluated on HumanEval, MBPP, and a new specialized dataset, LlmErrorEval. The tool achieved a 17.1% fix rate on LlmErrorEval, outperforming the state-of-the-art LLM-based debugger (LDB) by 8.9%. When integrated with LDB, the fix rate rose to 24.9%, demonstrating that rule-based fixing complements LLM-based debugging by handling structural "noise."
Crucially, LlmFix is highly efficient, taking an average of just 11.5ms to fix an error, whereas LLM-based debuggers require multiple model inferences taking seconds or minutes. Applying LlmFix as a post-processing step improved the Pass@1 accuracy of the 14 LLMs by an average of 7.5%.
Conclusion
LlmFix demonstrates that many LLM generation errors are structural and can be corrected using simple, deterministic rules. By addressing these "low-hanging fruit" errors, LlmFix provides a highly efficient post-processing step that significantly boosts the reliability of LLM-generated code without the need for expensive additional GPU cycles.
Paper Citation: Hao Wen, Yueheng Zhu, Chao Liu, Xiaoxue Ren, Weiwei Du, and Meng Yan.
- Fixing Function-Level Code Generation Errors for Foundation Large Language Models. ACM Trans. Softw. Eng. Methodol. 1, 1, Article 1 (August 2024).
Summary updated