TL;DR
As Large Language Models (LLMs) for code generation rapidly evolve through fine-tuning, merging, and new releases, they frequently introduce silent regressions in code quality and performance. To combat this, researchers have developed ReCatcher, the first systematic regression testing framework for Python code generation that evaluates updates across logical correctness, static code quality, and execution performance. Their findings reveal surprising degradations—such as GPT-4o regressing up to 50% in import handling compared to GPT-3.5-turbo.
What Problem Does This Paper Address?
When software engineers upgrade a traditional software library, they rely on regression testing to ensure nothing breaks. However, in the world of LLM-assisted programming, models are constantly updated via fine-tuning, model merging, or new version releases without a robust safety net.
Current LLM evaluation heavily over-indexes on functional correctness (using metrics like pass@k). This creates a massive blind spot: a new model version might pass the same functional tests but output code that is plagued with syntax errors, bloated with duplicate blocks, or drastically slower and more memory-intensive. Furthermore, traditional regression testing fails here because LLMs are inherently non-deterministic; the same prompt can yield different code structures across runs, making simple text diffs or basic unit tests insufficient.
Methods
The authors introduce ReCatcher, an end-to-end regression testing framework designed specifically to capture these multi-dimensional degradations.
As shown in Figure 1, the framework operates via a three-step pipeline:
- Code Generator: To handle LLM non-determinism, ReCatcher samples code snippets from both the baseline model ($LLM$) and the candidate update ($LLM'$) multiple times (default $m=10$) using a low temperature (0.1) for stability.
- Test Suite: The generated code is pushed through three distinct testing pillars:
- Logical Correctness: Validated via execution against benchmark unit tests.
- Static Code Issues: Analyzed using standard linters (Pylint) and token-based duplication detectors (PMD-CPD) mapped to a specialized taxonomy of LLM inefficiencies (e.g., syntax errors, missing imports, unnecessary conditional blocks).
- Performance Profiling: For code snippets that are functionally correct on both models, ReCatcher measures execution time and tracks memory allocation using Python’s
tracemallocmodule across multiple iterations ($n=5$). - Analyzer: Quantifies the percentage differences between the models and applies the non-parametric Mann-Whitney U test to determine if performance shifts are statistically significant.
Datasets
ReCatcher was evaluated using two prominent Python benchmarks:
- HumanEval+: An enhanced version of OpenAI's classic benchmark, consisting of 164 algorithmic problem-solving tasks.
- BigCodeBench: A more challenging, real-world benchmark containing 1,140 fine-grained tasks that require models to interact with 139 external libraries across 7 domains.
Key Insights and Ideas
- The "Smarter Model" Fallacy: Newer or more cost-effective models do not guarantee strict Pareto improvements. Upgrading to a model with higher general capabilities can severely degrade specific code-handling capabilities.
- Cross-Language Contamination: Fine-tuning an LLM on a dataset from one language (e.g., Kotlin) can inadvertently disrupt its internal syntax rules for another language (e.g., Python), leading to an influx of basic syntax errors.
- The Perils of Blind Model Merging: Merging a code-optimized model with a general-purpose foundation model (like Llama2) to boost general reasoning can severely dilute its coding proficiency, introducing massive regressions in both logic and code duplication.
- LLMs make Poor Performance Judges: When testing a baseline "LLM-as-a-Judge" approach (using GPT-4o-mini to guess which code sample performs better), the LLM achieved a very low F1-score (0.15) for execution time and memory. This underscores the necessity of ReCatcher's actual execution-based profiling over purely prompt-based evaluation.
Key Results
The authors stress-tested ReCatcher across three update scenarios (fine-tuning, merging, and new releases) using CodeLlama, DeepSeek-Coder, and the GPT family:
- Fine-Tuning: Models fine-tuned on cross-language datasets saw syntax errors spike by up to 12%, though they often showed minor improvements in maintainability (fewer duplicate comments).
- Merging: Merging CodeLlama with the general-purpose Llama2 caused a massive 18% regression in logical correctness on HumanEval+ and a 30.77% increase in syntax errors on BigCodeBench.
- Model Releases (GPT-3.5 vs GPT-4o): GPT-4o exhibited a staggering 51.64% regression in handling missing declarations and imports on BigCodeBench compared to GPT-3.5-turbo, likely due to difficulties navigating complex or deprecated external libraries.
- Model Releases (GPT-4o vs GPT-4o-mini): While GPT-4o-mini improved on logical correctness and import handling, it suffered a massive 80.49% performance degradation in execution time on HumanEval+ algorithmic tasks compared to its larger sibling.
Limitations and Future Work
While powerful, ReCatcher currently has a few limitations:
- Language Lock-in: It is exclusively tailored to Python.
- Cascading Failures: If a model update introduces a syntax error, ReCatcher cannot evaluate its logical correctness or performance for that run, potentially masking deeper logic regressions.
- Unit Test Dependency: The framework's performance and correctness metrics are only as good as the underlying benchmark's test suite coverage.
Future Directions: The authors plan to extend ReCatcher to support multi-language regression testing, integrate distribution-aware statistical analysis for more granular reporting, and factor in task difficulty levels to see exactly where models begin to fail during updates.
Summary updated