TL;DR
CodeLift is a novel reinforcement learning (RL) framework that transforms a standard software test suite into a dense, adaptive curriculum for training software engineering (SWE) agents. By dynamically reweighting tests based on their observed difficulty and assigning credit to intermediate progress through "snapshot slicing," CodeLift overcomes the sparsity of traditional binary rewards and the biases of simple pass-rate metrics, leading to significant performance gains on industry-standard benchmarks like SWE-bench.
What Problem Does This Paper Address?
Training LLM-based agents to solve real-world software issues is notoriously difficult. The standard "recipe" uses binary rewards: if an agent's code patch passes all hidden tests, it gets a reward of 1; otherwise, it gets 0. This is incredibly "sparse"—an agent that fixes 99% of a bug but fails one tiny edge case receives the same zero reward as an agent that writes complete gibberish.
A natural alternative is the test pass-rate (e.g., 80/100 tests passed = 0.8 reward). However, the authors identify three critical "imbalances" that make raw pass-rates unreliable:
- Multiplicity Imbalance: A single functionality might be covered by 100 trivial tests, while a complex core feature has only one. A patch that fixes the trivial tests but breaks the core feature would incorrectly receive a higher reward.
- Progress-Regression Imbalance: Most tests in a repo are "pass-to-pass" (P2P) regression tests. Only a few are "fail-to-pass" (F2P) tests that check the actual bug fix. Under simple pass-rates, the penalty for breaking one existing feature often outweighs the reward for fixing the bug, making the agent "afraid" to change code.
- Difficulty Imbalance: Easy tests dominate the reward signal, drowning out the learning signal from challenging, high-value tests.
Methods
CodeLift introduces two primary components to turn these imbalances into an advantage:
1. Behavior Balancing
Instead of treating every test as equal, CodeLift groups tests and patches based on their behavior. If a set of tests always fails or passes together across different patches, they are grouped. CodeLift then estimates the difficulty of each test group by looking at how many patch groups "kill" it.
- Adaptive Curriculum: As the agent improves and starts passing easier tests, those tests lose weight. The reward signal automatically shifts focus to the remaining "hard" tests that the agent still struggles with.
2. Snapshot Slicing
Agents often run tests themselves while working. CodeLift treats these "self-tests" as natural checkpoints. It evaluates these intermediate repository states (snapshots) against the "golden" test suite.
- Credit Assignment: If an agent makes great progress in the middle of a task but accidentally introduces a regression right at the end, standard RL would penalize the whole trajectory. Snapshot slicing allows CodeLift to identify and reward the successful intermediate steps, providing a much denser and more stable learning signal.
Datasets
The authors trained their models using:
- R2E-Gym: A procedural environment for scaling SWE agents.
- SWE-rebench v1: A decontaminated evaluation pipeline for SWE tasks.
Evaluation was performed on:
- SWE-bench Verified: 500 curated issues focusing on single-file tasks.
- SWE-bench Pro (Python subset): 266 issues involving more complex, multi-file changes.
Key Insights and Ideas
- The Curriculum is Already There: You don't need to manually design a curriculum for SWE agents. The inherent difficulty levels of the existing tests, when viewed through the lens of agent behavior, provide a natural progression from easy to hard.
- Conservative Bias is Real: Raw pass-rate rewards actually encourage agents to be too conservative. Because regression penalties are so high, agents often learn that the "safest" move is to change as little as possible. CodeLift's balancing brings the reward/penalty ratio close to parity (1.08x vs. the baseline's 3.71x).
- Stability Matters: Dense rewards are only useful if they are stable. CodeLift maintains a high "signal density" (75.1% active rollout groups) throughout training, whereas binary rewards often leave the agent with no signal at all when tasks get tough.
Key Results
CodeLift consistently outperformed both binary and enhanced pass-rate baselines across all metrics:
- SWE-bench Verified: Achieved a pass@1 of 38.30%, a significant jump over the Binary baseline (34.63%) and the best pass-rate variant (35.55%).
- SWE-bench Pro: Showed strong generalization to complex tasks, improving pass@1 to 20.16% compared to the Binary baseline's 16.54%.
- Robustness: The gains held true regardless of the underlying advantage estimator used (RLOO or GRPO), proving that the improvement comes from the reward design itself.
Limitations and Future Work
- Language Specificity: The current implementation and evaluation are focused on Python. While the principles are general, the engineering required to detect "self-tests" and run snapshots efficiently needs to be adapted for other languages (e.g., Java, C++).
- Computational Overhead: Snapshot slicing requires running the test suite on intermediate code states. While this runs on CPUs and doesn't require extra GPU time for rollout generation, it does increase the total "wall-clock" time for training.
- Future Directions: The authors suggest exploring how this "behavior-driven" curriculum could be applied to other complex reasoning tasks beyond software engineering, such as mathematical theorem proving or complex tool-use scenarios.
Summary updated