CYCLE: Learning to Self-Refine the Code Generation

Yangruibo Ding, Marcus J. Min, Gail Kaiser, Baishakhi Ray

Abstract

While Large Language Models (LLMs) have shown remarkable capabilities in one-shot code generation, they often struggle to correct their own errors when presented with execution feedback. This paper presents CYCLE, a framework that explicitly trains code LMs to self-refine. By leveraging a curated dataset of faulty code, execution feedback, and ground-truth fixes, CYCLE teaches models to iteratively debug and improve their output. Our results show that CYCLE boosts code generation performance by up to 63.5% relative to one-time generation, outperforming larger models through efficient iterative refinement.

Introduction

The standard paradigm for evaluating code LMs focuses on "one-time prediction" accuracy. However, real-world software development is inherently iterative. When a model generates a buggy snippet, developers expect it to be able to fix itself given a traceback or a failed assertion.

Current state-of-the-art models, including GPT-3.5, often fail in this "exploration mode." Instead of utilizing feedback to derive a new solution, these models frequently repeat their previous faulty code. This "copy-paste" behavior indicates a lack of inherent self-refinement capability, suggesting that scaling alone is insufficient for robust iterative programming.

Figure 1: GPT-3.5 failing to self-refine compared to the successful iterative correction by CYCLE.

The CYCLE Framework

CYCLE transforms code generation into an iterative process through three distinct phases:

  1. Phase I: Data Preparation: Using knowledge distillation, the authors identify model-specific weaknesses. A model is fine-tuned on correct code and then prompted to generate solutions. Failed attempts are collected to form "faulty code + execution feedback" pairs.
  2. Phase II: Learning to Refine: The model is trained on an aggregated input containing the problem description, the faulty code, and the execution results. To prevent the model from simply copying the faulty code, a Past Generation Mask (PGM) is introduced. This mask randomly hides tokens in the incorrect code during training, forcing the model to reconstruct the logic rather than relying on the previous output.
  3. Phase III: Iterative Self-Refinement: At inference time, the model enters a feedback loop. It generates code, executes it against unit tests, and feeds any failure data back into itself for a subsequent refinement attempt.
Figure 2: The three-phase architecture of the CYCLE framework: Data Preparation, Learning to Refine, and Iterative Self-Refinement.

Experimental Evaluation

The framework was evaluated on three major benchmarks: HumanEval, MBPP-Sanitized (MBPP-S), and APPS, covering a range from basic functions to complex algorithmic challenges. The training data consisted of approximately 71,000 samples distilled from CodeContest across Python, Java, and C++.

Key Insights

  • Self-refinement is not Emergent: Models up to 3B parameters do not naturally learn to self-refine from standard pre-training. Explicit training on error correction is necessary.
  • The "Copy-Paste" Trap: Without interventions like PGM, models treat the previous faulty code as a shortcut, leading to zero-edit "refinements."
  • Depth vs. Breadth: While traditional "Top-K" generation explores the breadth of the solution space, CYCLE explores its depth. These approaches are complementary and can be combined for superior results.
  • Efficiency: A CYCLE-trained 350M parameter model can outperform a 1B parameter StarCoder model, demonstrating that specialized training can compensate for smaller model scales.

Results

CYCLE consistently improved performance across all benchmarks:

  • Performance Gains: Relative improvements reached up to 63.5% over one-time generation.
  • Iterative Improvement: Unlike baseline models that plateau, CYCLE's accuracy continues to increase with each refinement step.
  • Feedback Sensitivity: Ablation studies showed that removing execution feedback significantly degrades performance, confirming that the model learns to interpret and act on error messages.

Conclusion

CYCLE demonstrates that self-refinement is a learnable skill for code LMs. By explicitly training on the relationship between faulty code, feedback, and corrections, models can move beyond the "copy-paste" trap. Future work includes extending this to more complex environments and exploring scenarios where models must generate their own test cases.

Summary updated

Abstract

Pre-trained code language models have achieved promising performance in code generation and improved the programming efficiency of human developers. However, their self-refinement capability is typically overlooked by the existing evaluations of code LMs, which focus only on the accuracy of the one-time prediction. For the cases when code LMs fail to implement the correct program, developers actually find it hard to debug and fix the faulty prediction since it is not written by the developers themselves. Unfortunately, our study reveals that code LMs cannot efficiently self-refine their faulty generations as well. In this paper, we propose CYCLE framework, learning to self-refine the faulty generation according to the available feedback, such as the execution results reported by the test suites. We evaluate CYCLE on three popular code generation benchmarks, HumanEval, MBPP, and APPS. The results reveal that CYCLE successfully maintains, sometimes improves, the quality of one-time code generation, while significantly improving the self-refinement capability of code LMs. We implement four variants of CYCLE with varied numbers of parameters across 350M, 1B, 2B, and 3B, and the experiments show that CYCLE consistently boosts the code generation performance, by up to 63.5
BibTeX
@article{ding2024cycle,
  author = {Yangruibo Ding and Marcus J. Min and Gail Kaiser and Baishakhi Ray},
  title = {{CYCLE: Learning to Self-Refine the Code Generation}},
  journal = {Proc. ACM Program. Lang.},
  year = {2024},
  doi = {10.1145/3649825},
}
Figure