SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, Boris Yangel

Abstract: LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.

Introduction

The race to build autonomous software engineering (SWE) agents is in full swing. While models like GPT-4 and DeepSeek-V3 show flashes of brilliance, they face a looming "data wall." High-quality, interactive training data is scarce, and the benchmarks used to measure progress—like the popular SWE-bench—are rapidly becoming contaminated as models are inadvertently trained on the very test cases meant to evaluate them. To address this, we present SWE-rebench, a framework designed to continuously harvest fresh, verified software engineering tasks from the wild.

The Automated Pipeline

The core of SWE-rebench is a four-stage automated pipeline designed to mine GitHub and create a massive dataset of over 21,000 interactive Python programming tasks. By automating the entire lifecycle of a task—from finding a bug report to configuring a containerized environment—the system ensures a steady supply of fresh data.

Overview of the automated pipeline for collecting and verifying software engineering data.
  1. Preliminary Filtering: The system mines GitHub and GHArchive for Pull Requests (PRs) linked to resolved issues in Python repositories.
  2. Automated Environment Setup: Using an LLM (Qwen2.5-72B), the pipeline scans repository metadata to "hypothesize" installation recipes, iteratively refining them based on error logs.
  3. Execution-based Verification: A task is only valid if the PR's original tests fail before the fix and pass after the fix (the "fail-to-pass" dynamic).
  4. LLM Labeling & Quality Assessment: Tasks are scored on clarity and complexity to ensure they are suitable for training.

Datasets and Benchmarks

Using this pipeline, we have constructed two primary resources:

  • SWE-rebench (The Dataset): A collection of 21,336 interactive Python tasks with verified environments, ideal for large-scale Reinforcement Learning.
  • SWE-rebench (The Benchmark): A curated, "decontaminated" subset of 294 tasks from issues created in early 2025—well after the knowledge cutoff of most current models.

Results and the Contamination Gap

The most striking insight from our evaluation is the "contamination gap." When models move from older benchmarks to fresh 2025 tasks, performance drops significantly. For instance, DeepSeek-V3-0324 solves nearly 40% of tasks on the older SWE-bench Verified but drops to approximately 21% on fresh tasks. This suggests that industry-standard performance metrics may be inflated by memorization.

Comparison of model performance on SWE-bench Verified vs. the fresh SWE-rebench (Mar–Apr) benchmark, highlighting the performance decay on newer tasks.

Our evaluation using a standardized ReAct scaffolding (Reasoning + Acting) shows that while GPT-4.1 (April 2025) leads the pack, DeepSeek-V3 remains the strongest open-source model. Interestingly, "thinking" variants of models did not always outperform "no-think" variants, suggesting that direct environment feedback is as critical as internal chain-of-thought reasoning for software engineering tasks.

Conclusion

By moving toward a continuous benchmarking model, SWE-rebench provides a more honest yardstick for the field. Future work will expand the pipeline beyond Python to include Go, Java, and JavaScript, further scaling the availability of high-quality, interactive SWE data.

Summary updated

Abstract

LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.
BibTeX
@inproceedings{badertdinov2025swe,
  author = {Ibragim Badertdinov and Alexander Golubev and Maksim Nekrashevich and Anton Shevtsov and Simon Karasik and Andrei Andriushchenko and Maria Trofimova and Daria Litvintseva and Boris Yangel},
  title = {{SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=nMpJoVmRy1},
}
Figure