When Old Meets New: Evaluating the Impact of Regression Tests on SWE Issue Resolution

Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, Martin Hirzel

Abstract

Large Language Model (LLM) agents have shown promise in automated software engineering (SWE), yet they are often overwhelmed by the scale of modern codebases and their associated test suites. While regression tests—existing tests that pass on a new bug—contain valuable information about correct system behavior, their volume typically exceeds LLM context windows and introduces significant noise. This paper introduces TestPrune, a framework designed to identify and select a "minimum viable test suite" for specific bug reports. By reducing test suites by over 1,000x, TestPrune enables more efficient and accurate bug reproduction and repair. Our evaluation on SWE-Bench shows that TestPrune improves issue resolution rates by up to 12.9% while reducing execution time by 27x, at a negligible cost of under $0.05 per issue.

Introduction: The Needle in the Haystack

The "needle in a haystack" problem is a significant hurdle for AI-driven software maintenance. Modern repositories contain thousands of files and extensive regression test suites. When a developer or an AI agent attempts to fix a newly reported bug, they must navigate this complexity. Traditionally, regression tests are overlooked during the fix process because they do not "fail" on the new issue. However, these tests serve as an implicit specification of the system, demonstrating correct function invocations and defining expected state transitions.

The challenge lies in the fact that feeding an entire test suite into an LLM is impractical due to context limits and the high cost of processing irrelevant data. Furthermore, large test suites often contain "flaky" tests or unrelated failures that can mislead an AI agent into generating incorrect patches. TestPrune addresses these issues by providing a focused, high-signal subset of tests that act as a "usage guide" for the agent.

Methods: The TestPrune Framework

TestPrune is a fully automated technique that prunes repository-wide test suites into a compact set of relevant tests. The process is divided into three main stages:

  1. Suspicious Function Localization: Utilizing the GitHub issue description, TestPrune employs an LLM to pinpoint the files and specific functions likely involved in the bug.
  2. Test Retrieval & Coverage Mapping: The framework identifies existing test files that exercise the localized "suspicious" functions. It utilizes line-level coverage tools to map exactly which tests touch the relevant code paths.
  3. Greedy Minimization: To ensure the resulting test set is as small as possible, TestPrune applies a greedy algorithm (Greedy-Additional). This algorithm selects the minimum number of tests required to cover all suspicious lines. In cases where multiple tests provide similar coverage, an LLM is used as a semantic tie-breaker to select the most contextually relevant test.
The TestPrune workflow: from issue description to a minimized set of regression tests used for reproduction and patch selection.

The selection process relies on carefully crafted prompts that allow the LLM to evaluate the relevance of potential test files against the bug report and the identified suspicious functions.

A sample prompt template used by TestPrune to rank and select the most relevant test files from a repository.

Experimental Evaluation

The effectiveness of TestPrune was evaluated using industry-standard benchmarks: SWE-Bench Lite and SWE-Bench Verified. These datasets consist of real-world GitHub issues from popular Python repositories. The experiments utilized state-of-the-art models, including GPT-4o and Claude 3.7-Sonnet, to drive the localization and repair agents.

Key Insights

  • Latent Information in Passing Tests: Even when a test passes, it provides the LLM with the correct syntax, parameter types, and environmental setup required for the functions it needs to modify. This significantly reduces the likelihood of the LLM generating "hallucinated" or syntactically incorrect code.
  • Precision over Volume: By reducing the average test suite size from thousands to approximately 9 tests, TestPrune achieves a precision of 0.63. This high signal-to-noise ratio prevents agents from being misled by irrelevant test outcomes.
  • Operational Efficiency: TestPrune is highly cost-effective, costing only a few cents per bug. The 27x reduction in execution time (from 24 minutes to under 1 minute) makes it a practical addition to continuous integration (CI) pipelines.

Results

The integration of TestPrune into existing SWE agent frameworks yielded substantial improvements:

  • Reproduction Success: When paired with the Otter framework, the rate of successful bug reproduction increased by 6.2% to 9.0%.
  • Repair Accuracy: Integrated into the Agentless repair pipeline, TestPrune boosted the final issue resolution rate by 9.4% to 12.9%.
  • Resource Savings: The framework achieved a massive 1,000x shrinkage in test suite size, drastically lowering the computational burden on the repair agents.

Conclusion

TestPrune demonstrates that providing LLMs with a small, highly relevant subset of existing regression tests is far more effective than providing no tests or an overwhelming number of them. By leveraging "old" tests to solve "new" problems, TestPrune enhances the precision and efficiency of automated software engineering agents. Future work will focus on extending this approach to other programming languages and exploring the use of minimized suites to guide the generation of new, targeted tests.

Summary updated

Abstract

Test suites in real-world projects are often large and achieve high code coverage, yet they remain insufficient for detecting all bugs. The abundance of unresolved issues in open-source project trackers highlights this gap. While regression tests are typically designed to ensure past functionality is preserved in the new version, they can also serve a complementary purpose: debugging the current version. Specifically, regression tests can (1) enhance the generation of reproduction tests for newly reported issues, and (2) validate that patches do not regress existing functionality. We present TestPrune, a fully automated technique that leverages issue tracker reports and strategically reuses regression tests for both bug reproduction and patch validation. A key contribution of TestPrune is its ability to automatically minimize the regression suite to a small, highly relevant subset of tests. Due to the predominance of LLM-based debugging techniques, this minimization is essential as large test suites exceed context limits, introduce noise, and inflate inference costs. TestPrune can be plugged into any agentic bug repair pipeline and orthogonally improve overall performance. As a proof of concept, we show that TestPrune leads to a 6.2%-9.0% relative increase in issue reproduction rate within the Otter framework and a 8.0%-12.9% relative increase in issue resolution rate within Agentless, SWE-Agent, and Trae agent on SWE-Bench Lite and SWE-Bench Verified benchmarks. Compared to the benefits, the model API cost overhead of TestPrune is minimal, at $0.02 and $0.05 per SWE-Bench instance using GPT-4o and Claude-3.7-Sonnet models, respectively.
BibTeX
@misc{chen2025when,
  author = {Yang Chen and Toufique Ahmed and Reyhaneh Jabbarvand and Martin Hirzel},
  title = {{When Old Meets New: Evaluating the Impact of Regression Tests on SWE Issue Resolution}},
  year = {2025},
  eprint = {2510.18270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.18270},
}
Figure