In the race to build autonomous software engineering agents, Reinforcement Learning (RL) has become the tool of choice. However, a major hurdle remains: RL baselines for these "long-horizon" tasks are notoriously finicky and hard to reproduce. When a new paper claims a breakthrough, is it because of a brilliant new algorithm, or simply because the baseline it was compared against was poorly tuned?
Arctic-SWE tackles this "reproducibility crisis" head-on. Instead of chasing a flashy new algorithm, the researchers at Snowflake and their collaborators focused on model-system co-design. By treating the training infrastructure and the model as a single, integrated unit, they’ve established a new, high-bar baseline that achieves state-of-the-art results with a fraction of the compute used by previous methods.
TL;DR
Arctic-SWE is a fully asynchronous RL framework that stabilizes long-horizon agent training through rigorous model-system co-design. By fixing hidden instabilities in numerical precision, state management, and data curation, it allows a 4B-parameter model to match much larger, more complex systems on the SWE-bench benchmark using 6x fewer training rollouts.
What Problem Does This Paper Address?
Training an agent to solve a GitHub issue is a "long-horizon" task—the agent must explore a repository, edit code, and run tests over many steps. Standard RL often fails here because of instability.
In asynchronous RL, where the model is constantly generating new experiences while simultaneously being updated, several "silent killers" emerge:
- Policy Lag: The "worker" models generating data might be several versions behind the "trainer" model.
- Numerical Drift: Tiny differences in how numbers are rounded between the inference engine and the training engine can compound over hundreds of steps, poisoning the learning signal.
- Stale State: If a model's internal "memory" (KV cache) isn't perfectly reset after a weight update, it starts hallucinating based on its old self.
Methods
The authors didn't invent a new RL algorithm; they used CISPO (a variant of PPO). Their contribution lies in the "recipe" and the "testbed."
- The Overfitting Stress Test: To find bugs in their system, they forced the model to overfit on a tiny pool of 160 solvable tasks. If the model couldn't maintain 100% success on this small set, the system was broken.
- System Fixes: They aligned the numerical paths (using FP32 for critical recurrent states) and enforced a total cache rebuild after every weight refresh.
- Gated Rewards & Raw Token Checks: They realized that standard "parsers" are too forgiving. If an agent produces a malformed command that the parser "fixes," the agent is accidentally rewarded for bad syntax. Arctic-SWE implements a "Gate" that zeros out rewards for any format violation.
- Data Curation Pipeline: They audited 21,000 tasks from existing datasets, using an LLM "judge" to ensure that the issue description, the unit tests, and the "gold" solution actually agreed with each other.
Datasets
The researchers utilized several key datasets for training and evaluation:
- Training: A curated subset of R2E-Gym, SWE-rebench V2, and Scale-SWE.
- Evaluation: SWE-bench Verified (a high-quality subset of 483 problems) and SWE-bench Pro (a more challenging set involving multiple versions and repositories).
Key Insights and Ideas
- Early Success is a Trap: As shown in Figure 1, standard RL recipes (like Prime-RL or SkyRL) often show a beautiful upward reward curve before suddenly collapsing. Without the "overfitting stress test," you might stop training early and never realize your recipe is unstable.
- System Details ARE Algorithmic Details: In async RL, the speed of your inference engine and how you handle your KV cache aren't just "engineering details"—they fundamentally change the distribution of data the optimizer sees.
- The Forgetting Curve: The authors discovered a "capacity" issue. While a 9B model can retain knowledge of tasks it has already mastered, a 4B model starts "forgetting" old tasks as it tries to learn new, harder ones.
- Test Strengthening Matters: Many existing benchmark tests are "too loose," allowing agents to pass by "cheating" (e.g., hardcoding a specific filename). By automatically strengthening these tests, the authors improved the agent's ability to generalize to unseen problems.
Key Results
Arctic-SWE sets a formidable new baseline for the community:
- Efficiency: On raw data, their Qwen3.5-4B model achieved a 62.0% pass@1 on SWE-bench Verified. This matches the performance of FrogNano, but Arctic-SWE required only one-sixth as many training rollouts (approx. 43k vs 256k).
- State-of-the-Art for Small Models: With their curated data, the 4B model reached 65.0% on Verified and jumped from 37.9% to 59.7% on the difficult SWE-bench Pro-v2.
- Scaling: The same recipe pushed a Qwen3.5-9B model to 70.2% on SWE-bench Pro-v2.
Limitations and Future Work
- The 4B Ceiling: The 4B model's tendency to forget mastered tasks suggests that for very small models, we may need better "replay" buffers or different objective functions to maintain performance over long training runs.
- Single-Seed Results: Most comparisons were done with a single seed due to compute constraints, though the authors used 21-update moving averages to ensure the trends were real.
- Language Scope: The current training data is focused on Python. Expanding this to a multi-lingual setting is a natural next step.
Arctic-SWE serves as a powerful reminder that in the world of RL, a stable foundation is worth more than a fancy new algorithm. By documenting these "missing pieces" of the training recipe, the authors have provided a clear starting point for the next generation of AI agents.
Summary updated