TL;DR
This paper explores how to move beyond human-in-the-loop grading for code generation by using execution feedback—compiler errors and test results—as the primary "teacher." By combining targeted repair curricula with Group-Relative Policy Optimization (GRPO), the authors significantly improved a base model's (GLM-4.7-Flash) ability to not only write correct C++ code on the first try but also to effectively self-correct when given execution feedback.
What Problem Does This Paper Address?
Training Large Language Models (LLMs) for coding usually relies on Supervised Fine-Tuning (SFT) on massive datasets of "correct" code. However, this approach has a major blind spot: it doesn't teach the model why code fails or how to fix it. Human feedback is the traditional solution, but it is expensive, slow, and difficult to scale.
The authors address the challenge of making model improvement repeatable and grounded in reality. In the "real world," code isn't judged by how much it looks like a training sample; it's judged by whether it compiles and passes its tests. This paper investigates an execution-guided workflow that uses the compiler and test suite as automated "verifiers" to provide structured rewards for Reinforcement Learning (RL).
Methods
The researchers developed a multi-stage post-training pipeline:
- Task Construction & Repair Curricula: Instead of just providing solutions, they built "repair tasks." For example, if a model fails because of a naming collision in C++, a specific training task is generated that includes the faulty code and the compiler's diagnostic error, forcing the model to practice the fix.
- SFT Warm Start: The model (GLM-4.7-Flash) is first fine-tuned on full solutions and prepared repair examples to establish a baseline of "good behavior."
- Group-Relative Policy Optimization (GRPO): This is the core RL component. The model generates multiple candidate solutions for a prompt. These are compiled and tested. GRPO then updates the model by comparing the relative rewards of these candidates—essentially "reinforcing" the versions that passed more tests compared to their peers in the same batch.
- Structured Rewards: Rather than a simple binary "pass/fail," the authors used verifiers that produce granular rewards. For a phone-number validation task, they used 12 distinct boolean checks (e.g., handling country codes, invalid characters, etc.). The reward is calculated based on how many of these specific "kernels" the code satisfies.
- Whole-File Editing: The model is trained to return the entire content of a file, which simplifies the integration with standard build tools and compilers.
Datasets
- Training Data: The authors used a mix of datasets, including 1,117 Aider-format examples for SFT and a "Synth v1" set of 260 examples. They also created targeted curricula for specific C++ failure mechanisms (like the "Phone Number" validation task).
- Evaluation (Fixed26): A project-specific benchmark of 26 C++ coding tasks.
- Note on Overlap: The authors transparently note that their strongest configuration ("Generalized C++") included six tasks in the evaluation set that were also present in the training set, and they used a "best-four" cohort selection for reporting.
Key Insights and Ideas
- Execution as Ground Truth: By grounding rewards in execution, the model learns the "physics" of the programming language. A compiler error is an objective truth that provides a much stronger signal for reinforcement than a human's subjective preference for coding style.
- The "Repair" Skill is Distinct: The paper highlights that a model's ability to solve a task initially and its ability to repair a failed attempt are different skills. Targeted repair practice (training on
faulty_code + error_msg -> fixed_code) is essential for high "conditional recovery" rates. - Infrastructure Error Handling: A clever engineering detail is how they handle "infrastructure errors" (e.g., a test harness failure). Instead of penalizing the model for a system glitch, they assign these samples the mean score of the valid samples in their group, preventing noisy data from corrupting the RL gradients.
Key Results
The improvements over the base model were dramatic:
- First-Round Success: The base GLM-4.7-Flash model solved only 1.9% of tasks on the first attempt. The best execution-trained configuration reached 45.2%.
- Two-Round Completion: When allowed one round of feedback (the model sees its own error and tries again), the success rate climbed to 61.5%.
- Conditional Recovery: This metric measures how often a model fixes a first-round failure. The execution-guided RL models showed a significant jump in this ability, proving that the "repair curricula" actually taught the model how to use feedback.
- Task-Level Shifts: Interestingly, the training wasn't a "rising tide" for all tasks. While success on the "Phone Number" task rose from 1/4 to 4/4 trials, some other tasks (like "Gigasecond") actually saw regressions, suggesting that optimizing for specific behaviors can sometimes lead to forgetting others.
Limitations and Future Work
- Small Benchmark: The "Fixed26" suite is relatively small. The authors acknowledge the need to test these interventions on larger, fresher task families.
- Prompt Sensitivity: The results were achieved using "evolving prompt contracts" (providing the model with specific API/build info derived from tests). This suggests the model's performance is still highly dependent on how the task is framed.
- Manual Intervention: While the grading is automated, the selection of which failure to target next in the curriculum still requires a researcher to review failures and design the next intervention.
- Future Direction: The authors suggest automating the diagnosis and selection of training interventions, effectively creating a fully autonomous "self-teaching" loop for coding models.
Summary updated