SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, Boris Yangel

TL;DR

SWE-rebench introduces a fully automated pipeline that continuously harvests real-world software engineering tasks from GitHub to combat data scarcity and benchmark contamination. The authors release a massive dataset of over 21,000 interactive Python tasks optimized for reinforcement learning, alongside a live, decontaminated benchmark that reveals how data leakage may be artificially inflating current LLM performance scores.

What Problem Does This Paper Address?

While LLM-based agents are rapidly evolving to handle complex software engineering (SWE) tasks, advancing the field is blocked by two major hurdles:

  1. Data Scarcity for Interactive Learning: Training robust agents via Reinforcement Learning (RL) requires environments where agents can execute code, observe feedback, and iterate. Existing datasets are typically small, static, or limited to one-shot code generation.
  2. Benchmark Contamination: Popular static benchmarks like SWE-bench have been public since late 2023. Newer LLMs are frequently exposed to these exact test repositories during pre-training or post-training, leading to inflated evaluation scores that reflect memorization rather than true generalization.

Methods

The authors design a scalable, four-stage automated pipeline to continuously mine and verify tasks without human intervention:

  1. Preliminary Task Collection: Automatically scrapes GitHub Archive and active repositories for resolved issues linked to merged pull requests (PRs) in Python codebases with permissive licenses.
  2. Automated Installation Configuration: Instead of relying on manual environment setups, the pipeline uses an "agentless" LLM approach (powered by Qwen2.5-72B-Instruct) to scan repository files and generate structured JSON installation recipes.
  3. Execution-Based Verification: Using a distributed computing platform (TractoAI), the pipeline builds isolated containers to verify that the test patch fails before the solution is applied and passes afterwards. Exact dependency versions are pinned to guarantee future reproducibility.
  4. Automated Instance Quality Assessment: To filter out poorly specified or unsolvable tasks, the authors fine-tune an LLM on human annotations from SWE-bench Verified to predict issue clarity, task complexity, and test correctness.
Figure 1: Overview of the automated pipeline for collecting software engineering data.

Datasets

  • SWE-rebench Dataset: A public dataset containing 21,336 verifiable, interactive Python SWE tasks spanning 3,468 distinct GitHub repositories, specifically designed for training agents via reinforcement learning at scale.
  • SWE-rebench Benchmark: A curated, contamination-free evaluation subset consisting of 294 executable tasks from 169 repositories, restricted strictly to issues created in 2025 (post-dating the training cutoffs of most models).

Key Insights and Ideas

  • The Contamination Tax is Real: Comparing performance on older benchmarks versus fresh 2025 tasks reveals a noticeable performance drop for several top-tier models, strongly indicating that static benchmark scores are inflated due to data exposure.
  • Reasoning Tokens Aren't a Silver Bullet for Scaffolding: Interestingly, Qwen3 models performed similarly on these interactive tasks whether "think mode" (explicit reasoning tokens) was enabled or disabled. This suggests that strong base capabilities matter more than explicit test-time reasoning steps when operating inside a ReAct scaffolding.
  • High Stochasticity Demands Multiple Runs: Agent behavior is highly variable. The authors run each model 5 times to compute a standard error of the mean (SEM), highlighting that evaluating an agent only once or reporting only the "best-of-N" run risks severely overstating an LLM's actual capabilities.

Key Results

The authors evaluated several frontier open- and closed-source models using a standardized, minimal ReAct scaffolding.

Table 1: Comparison of model performance on SWE-rebench Jan 2025 and SWE-rebench (Mar–Apr 2025). All metrics are reported in percentages. Models released after 1st of March 2025 are denoted with an asterisk (*).
  • Top Performer: gpt-4.1-2025-04-14 achieved the highest resolution rate on the March–April 2025 slice at 26.7%, though it showed a noticeable decline compared to its performance on the January subset (31.1%).
  • Open-Source Leaders: DeepSeek-V3 models demonstrated the strongest open-source performance, with DeepSeek-V3-1226 resolving 21.9% of the March–April tasks.
  • Evidence of Contamination: As shown in the paper's comparison table, DeepSeek-V3-0324 scores 39.7% on the older SWE-bench Verified but drops sharply to 21.3% on the fresh SWE-rebench tasks, underscoring the necessity of dynamic, time-restricted benchmarking.

Limitations and Future Work

  • Language Constraints: The initial release of the pipeline and dataset is exclusively limited to Python. Adapting the pipeline to support other major language ecosystems (like Go, Java, Rust, or JavaScript) will require building language-specific test parsers and environment builders.
  • Lower Absolute Success Rates: Because the pipeline is fully automated, some tasks might remain imperfectly described or unsolvable solely from the issue description, leading to lower absolute success rates compared to meticulously hand-curated benchmarks.
  • Future Directions: The authors plan to expand data volume by moving beyond issue-linked PRs to arbitrary code changes, and intend to continuously update the public leaderboard with fresh evaluation slices.

Summary updated

Abstract

LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.
BibTeX
@inproceedings{badertdinov2025swe,
  author = {Ibragim Badertdinov and Alexander Golubev and Maksim Nekrashevich and Anton Shevtsov and Simon Karasik and Andrei Andriushchenko and Maria Trofimova and Daria Litvintseva and Boris Yangel},
  title = {{SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=nMpJoVmRy1},
}
Figure