TL;DR
SWE-rebench introduces a fully automated pipeline that continuously harvests real-world software engineering tasks from GitHub to combat data scarcity and benchmark contamination. The authors release a massive dataset of over 21,000 interactive Python tasks optimized for reinforcement learning, alongside a live, decontaminated benchmark that reveals how data leakage may be artificially inflating current LLM performance scores.
What Problem Does This Paper Address?
While LLM-based agents are rapidly evolving to handle complex software engineering (SWE) tasks, advancing the field is blocked by two major hurdles:
- Data Scarcity for Interactive Learning: Training robust agents via Reinforcement Learning (RL) requires environments where agents can execute code, observe feedback, and iterate. Existing datasets are typically small, static, or limited to one-shot code generation.
- Benchmark Contamination: Popular static benchmarks like SWE-bench have been public since late 2023. Newer LLMs are frequently exposed to these exact test repositories during pre-training or post-training, leading to inflated evaluation scores that reflect memorization rather than true generalization.
Methods
The authors design a scalable, four-stage automated pipeline to continuously mine and verify tasks without human intervention:
- Preliminary Task Collection: Automatically scrapes GitHub Archive and active repositories for resolved issues linked to merged pull requests (PRs) in Python codebases with permissive licenses.
- Automated Installation Configuration: Instead of relying on manual environment setups, the pipeline uses an "agentless" LLM approach (powered by Qwen2.5-72B-Instruct) to scan repository files and generate structured JSON installation recipes.
- Execution-Based Verification: Using a distributed computing platform (TractoAI), the pipeline builds isolated containers to verify that the test patch fails before the solution is applied and passes afterwards. Exact dependency versions are pinned to guarantee future reproducibility.
- Automated Instance Quality Assessment: To filter out poorly specified or unsolvable tasks, the authors fine-tune an LLM on human annotations from SWE-bench Verified to predict issue clarity, task complexity, and test correctness.
Datasets
- SWE-rebench Dataset: A public dataset containing 21,336 verifiable, interactive Python SWE tasks spanning 3,468 distinct GitHub repositories, specifically designed for training agents via reinforcement learning at scale.
- SWE-rebench Benchmark: A curated, contamination-free evaluation subset consisting of 294 executable tasks from 169 repositories, restricted strictly to issues created in 2025 (post-dating the training cutoffs of most models).
Key Insights and Ideas
- The Contamination Tax is Real: Comparing performance on older benchmarks versus fresh 2025 tasks reveals a noticeable performance drop for several top-tier models, strongly indicating that static benchmark scores are inflated due to data exposure.
- Reasoning Tokens Aren't a Silver Bullet for Scaffolding: Interestingly, Qwen3 models performed similarly on these interactive tasks whether "think mode" (explicit reasoning tokens) was enabled or disabled. This suggests that strong base capabilities matter more than explicit test-time reasoning steps when operating inside a ReAct scaffolding.
- High Stochasticity Demands Multiple Runs: Agent behavior is highly variable. The authors run each model 5 times to compute a standard error of the mean (SEM), highlighting that evaluating an agent only once or reporting only the "best-of-N" run risks severely overstating an LLM's actual capabilities.
Key Results
The authors evaluated several frontier open- and closed-source models using a standardized, minimal ReAct scaffolding.
- Top Performer:
gpt-4.1-2025-04-14achieved the highest resolution rate on the March–April 2025 slice at 26.7%, though it showed a noticeable decline compared to its performance on the January subset (31.1%). - Open-Source Leaders: DeepSeek-V3 models demonstrated the strongest open-source performance, with
DeepSeek-V3-1226resolving 21.9% of the March–April tasks. - Evidence of Contamination: As shown in the paper's comparison table,
DeepSeek-V3-0324scores 39.7% on the older SWE-bench Verified but drops sharply to 21.3% on the fresh SWE-rebench tasks, underscoring the necessity of dynamic, time-restricted benchmarking.
Limitations and Future Work
- Language Constraints: The initial release of the pipeline and dataset is exclusively limited to Python. Adapting the pipeline to support other major language ecosystems (like Go, Java, Rust, or JavaScript) will require building language-specific test parsers and environment builders.
- Lower Absolute Success Rates: Because the pipeline is fully automated, some tasks might remain imperfectly described or unsolvable solely from the issue description, leading to lower absolute success rates compared to meticulously hand-curated benchmarks.
- Future Directions: The authors plan to expand data volume by moving beyond issue-linked PRs to arbitrary code changes, and intend to continuously update the public leaderboard with fresh evaluation slices.
Summary updated