ReCatcher: Towards LLMs Regression Testing for Code Generation

Altaf Allah Abbassi, Leuson Da Silva, Amin Nikanjam, Foutse KhomharXiv 2025 · arXiv

TL;DR

As Large Language Models (LLMs) for code generation rapidly evolve through fine-tuning, merging, and new releases, they frequently introduce silent regressions in code quality and performance. To combat this, researchers have developed ReCatcher, the first systematic regression testing framework for Python code generation that evaluates updates across logical correctness, static code quality, and execution performance. Their findings reveal surprising degradations—such as GPT-4o regressing up to 50% in import handling compared to GPT-3.5-turbo.

What Problem Does This Paper Address?

When software engineers upgrade a traditional software library, they rely on regression testing to ensure nothing breaks. However, in the world of LLM-assisted programming, models are constantly updated via fine-tuning, model merging, or new version releases without a robust safety net.

Current LLM evaluation heavily over-indexes on functional correctness (using metrics like pass@k). This creates a massive blind spot: a new model version might pass the same functional tests but output code that is plagued with syntax errors, bloated with duplicate blocks, or drastically slower and more memory-intensive. Furthermore, traditional regression testing fails here because LLMs are inherently non-deterministic; the same prompt can yield different code structures across runs, making simple text diffs or basic unit tests insufficient.

Methods

The authors introduce ReCatcher, an end-to-end regression testing framework designed specifically to capture these multi-dimensional degradations.

Fig. 1. ReCatcher Architecture: LLM Regression Testing Framework for Code Generation

As shown in Figure 1, the framework operates via a three-step pipeline:

  1. Code Generator: To handle LLM non-determinism, ReCatcher samples code snippets from both the baseline model ($LLM$) and the candidate update ($LLM'$) multiple times (default $m=10$) using a low temperature (0.1) for stability.
  2. Test Suite: The generated code is pushed through three distinct testing pillars:
  3. Logical Correctness: Validated via execution against benchmark unit tests.
  4. Static Code Issues: Analyzed using standard linters (Pylint) and token-based duplication detectors (PMD-CPD) mapped to a specialized taxonomy of LLM inefficiencies (e.g., syntax errors, missing imports, unnecessary conditional blocks).
  5. Performance Profiling: For code snippets that are functionally correct on both models, ReCatcher measures execution time and tracks memory allocation using Python’s tracemalloc module across multiple iterations ($n=5$).
  6. Analyzer: Quantifies the percentage differences between the models and applies the non-parametric Mann-Whitney U test to determine if performance shifts are statistically significant.

Datasets

ReCatcher was evaluated using two prominent Python benchmarks:

  • HumanEval+: An enhanced version of OpenAI's classic benchmark, consisting of 164 algorithmic problem-solving tasks.
  • BigCodeBench: A more challenging, real-world benchmark containing 1,140 fine-grained tasks that require models to interact with 139 external libraries across 7 domains.

Key Insights and Ideas

  • The "Smarter Model" Fallacy: Newer or more cost-effective models do not guarantee strict Pareto improvements. Upgrading to a model with higher general capabilities can severely degrade specific code-handling capabilities.
  • Cross-Language Contamination: Fine-tuning an LLM on a dataset from one language (e.g., Kotlin) can inadvertently disrupt its internal syntax rules for another language (e.g., Python), leading to an influx of basic syntax errors.
  • The Perils of Blind Model Merging: Merging a code-optimized model with a general-purpose foundation model (like Llama2) to boost general reasoning can severely dilute its coding proficiency, introducing massive regressions in both logic and code duplication.
  • LLMs make Poor Performance Judges: When testing a baseline "LLM-as-a-Judge" approach (using GPT-4o-mini to guess which code sample performs better), the LLM achieved a very low F1-score (0.15) for execution time and memory. This underscores the necessity of ReCatcher's actual execution-based profiling over purely prompt-based evaluation.

Key Results

The authors stress-tested ReCatcher across three update scenarios (fine-tuning, merging, and new releases) using CodeLlama, DeepSeek-Coder, and the GPT family:

  • Fine-Tuning: Models fine-tuned on cross-language datasets saw syntax errors spike by up to 12%, though they often showed minor improvements in maintainability (fewer duplicate comments).
  • Merging: Merging CodeLlama with the general-purpose Llama2 caused a massive 18% regression in logical correctness on HumanEval+ and a 30.77% increase in syntax errors on BigCodeBench.
  • Model Releases (GPT-3.5 vs GPT-4o): GPT-4o exhibited a staggering 51.64% regression in handling missing declarations and imports on BigCodeBench compared to GPT-3.5-turbo, likely due to difficulties navigating complex or deprecated external libraries.
  • Model Releases (GPT-4o vs GPT-4o-mini): While GPT-4o-mini improved on logical correctness and import handling, it suffered a massive 80.49% performance degradation in execution time on HumanEval+ algorithmic tasks compared to its larger sibling.

Limitations and Future Work

While powerful, ReCatcher currently has a few limitations:

  • Language Lock-in: It is exclusively tailored to Python.
  • Cascading Failures: If a model update introduces a syntax error, ReCatcher cannot evaluate its logical correctness or performance for that run, potentially masking deeper logic regressions.
  • Unit Test Dependency: The framework's performance and correctness metrics are only as good as the underlying benchmark's test suite coverage.

Future Directions: The authors plan to extend ReCatcher to support multi-language regression testing, integrate distribution-aware statistical analysis for more granular reporting, and factor in task difficulty levels to see exactly where models begin to fail during updates.

Summary updated

Abstract

Large Language Models (LLMs) for code generation evolve rapidly through fine-tuning, merging, or new model releases. However, such updates can introduce regressions, not only in correctness but also in code quality and performance. To address this, we present ReCatcher, a regression testing framework for Python code generation. ReCatcher systematically compares two LLMs, typically a current model and a candidate update, across three dimensions: logical correctness, static code quality, and execution performance. We apply ReCatcher to assess regressions across three update scenarios, fine-tuning, merging, and model release, using CodeLlama, DeepSeek-Coder, and GPT-4o. Our evaluation shows that fine-tuning with cross-language datasets increases syntax errors by up to 12%. Merging with general-purpose models like Llama2 leads to regressions in correctness by up to 18%. GPT-4o introduces regressions of up to 50% in handling missing imports compared to GPT-3.5-turbo, while GPT-4o-mini suffers up to 80% performance degradation in execution time versus GPT-4o. Overall, logical correctness, performance, and error handling (e.g., syntax errors and missing imports) are the most regression-prone areas. Comparing ReCatcher with baseline solutions, it presents better and consistent accuracy across logical and performance aspects. ReCatcher highlights the importance of systematic regression evaluation before adopting new models, while assisting researchers and practitioners in making more informed update decisions.
BibTeX
@misc{abbassi2025recatcher,
  author = {Altaf Allah Abbassi and Leuson Da Silva and Amin Nikanjam and Foutse Khomh},
  title = {{ReCatcher: Towards LLMs Regression Testing for Code Generation}},
  year = {2025},
  eprint = {2507.19390},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.19390},
}
Figure