Abstract
Automated unit test generation is a critical component of modern software development, yet evaluating the true capabilities of Large Language Models (LLMs) in this domain remains challenging. Existing benchmarks often suffer from data contamination and structural simplicity, leading to an overestimation of model performance. This paper introduces UnLeakedTestbench (ULT), a rigorous benchmark comprising 3,909 clean, high-complexity Python functions. By comparing performance across ULT and its contaminated counterpart, PreLeakedTestbench (PLT), we demonstrate a significant "memorization gap." Our findings reveal that while LLMs perform well on familiar or simple code, their ability to reason through novel, complex logic is substantially lower than previously reported.
1. Introduction
The promise of LLMs in software engineering lies in their ability to automate repetitive tasks, with unit test generation being a primary target. However, current evaluation frameworks face two major hurdles. First, data contamination occurs when models are trained on the same open-source repositories used for benchmarking, allowing them to recall existing tests rather than generate them through logic. Second, many benchmarks rely on structurally simple code, such as competitive programming problems, which do not reflect the messy, high-complexity nature of real-world production software.
To address these issues, we present ULT, a benchmark specifically curated to test LLMs against non-trivial, uncontaminated code. By filtering for high cyclomatic complexity and strictly removing leaked data, ULT provides a more accurate measure of an LLM's generalization and reasoning capabilities.
2. Methodology
2.1 Benchmark Curation
The construction of ULT involved a multi-stage pipeline designed to ensure both difficulty and cleanliness:
- Sourcing: Candidate functions were extracted from The Stack v2, a large-scale corpus of permissively licensed code.
- Complexity Filtering: To move beyond "toy" programs, we only included functions with a cyclomatic complexity of 10 or higher. This ensures the code contains multiple nested loops and conditional branches.
- Verification: Functions were filtered for self-containment. We utilized GPT-4o to verify and, where necessary, repair functions to ensure they were executable and testable in isolation.
2.2 Decontamination Strategy
To isolate the effects of memorization, we searched The Stack v2 for any existing test definitions or assertions associated with the curated functions.
- ULT (UnLeakedTestbench): Functions for which no existing tests were found in the training corpus.
- PLT (PreLeakedTestbench): A set of 18,169 functions where tests were present in the training data, serving as a control group to measure the "memorization boost."
2.3 Evaluation Metrics
We evaluated models using execution-based metrics, including Pass@1 (accuracy), Branch Coverage (the percentage of logic paths executed), and Mutation Scores (the ability of the generated tests to detect injected bugs).
3. Key Insights and Results
3.1 The Performance Cliff
The results indicate a stark disparity between performance on traditional benchmarks and the uncontaminated ULT. For example, LLMs averaged a 12.69% Pass@1 on ULT, compared to 57.74% on the popular TestEval benchmark. Branch coverage also saw a dramatic decline, falling from 82.04% on TestEval to just 30.22% on ULT.
Even state-of-the-art models are not immune. Qwen2.5-Coder-32B, which achieves nearly 78% accuracy on TestEval, saw its performance plummet to 17.83% when faced with the novel tasks in ULT.
3.2 Memorization vs. Reasoning
Our analysis confirms that high scores on older benchmarks are often a byproduct of memorization. Performance on ULT correlates strongly ($r = 0.79$) with general coding ability (as measured by BigCodeBench), whereas the correlation on contaminated benchmarks is much weaker. This suggests that ULT is a more reliable proxy for a model's true ability to generalize to proprietary or new codebases.
3.3 The Impact of Complexity
Structural complexity serves as a significant barrier for current LLMs. As cyclomatic complexity increases, branch coverage drops significantly. While models are relatively proficient at testing simple linear logic, they struggle to navigate the deeply nested conditional structures common in real-world software.
4. Discussion and Limitations
While ULT provides a more rigorous evaluation than previous benchmarks, it is currently limited to Python. LLM performance may vary across different programming languages based on syntax and training data density. Furthermore, this study focuses on function-level unit tests. Real-world development requires class-level and integration testing, which involve external dependencies and mocking—complexities not yet captured in this framework.
5. Conclusion
The introduction of ULT highlights a significant gap between LLM memorization and true logical reasoning in software testing. By providing a benchmark that is both complex and uncontaminated, we offer a more realistic assessment of current AI capabilities. Future work will focus on expanding ULT to multiple languages and addressing the "oracle problem" to better evaluate the semantic correctness and readability of AI-generated tests.