Abstract
Unit testing is crucial for software development and maintenance. Effective unit testing ensures and improves software quality, but writing unit tests is time-consuming and labor-intensive. Recent studies have proposed deep learning (DL) techniques or large language models (LLMs) to automate unit test generation. These models are usually trained or fine-tuned on large-scale datasets. Despite growing awareness of the importance of data quality, there has been limited research on the quality of datasets used for test generation. To bridge this gap, we systematically examine the impact of noise on the performance of learning-based test generation models. We first apply the open card sorting method to analyze the most popular and largest test generation dataset, Methods2Test, to categorize eight distinct types of noise. Further, we conduct detailed interviews with 17 domain experts to validate and assess the importance, reasonableness, and correctness of the noise taxonomy. Then, we propose CleanTest, an automated noise-cleaning framework designed to improve the quality of test generation datasets. CleanTest comprises three filters: a rule-based syntax filter, a rule-based relevance filter, and a model-based coverage filter. To evaluate its effectiveness, we apply CleanTest on two widely-used test generation datasets, i.e., Methods2Test and Atlas. Our findings indicate that 43.52% and 29.65% of datasets contain noise, highlighting its prevalence. Finally, we conduct comparative experiments using four LLMs (i.e., CodeBERT, AthenaTest, StarCoder, and CodeLlama7B) to assess the impact of noise on test generation performance. The results show that filtering noise positively influences the test generation ability of the models.
Introduction
Automated unit test generation is a cornerstone of modern software engineering, promising to reduce the manual effort required to ensure code reliability. Current state-of-the-art approaches rely on fine-tuning Large Language Models (LLMs) on massive datasets mined from open-source repositories like GitHub. However, these datasets often prioritize quantity over quality. If a model is trained on "noisy" data—such as tests that are syntactically broken or irrelevant to the code under test—it learns to generate poor-quality tests. This research systematically categorizes this noise and introduces a framework to mitigate its impact.
Taxonomy of Noise and Methodology
To understand the nature of data quality in this domain, the researchers performed a manual inspection of 50,000 pairs of focal methods and test cases from the Methods2Test dataset using "open card sorting." This process identified eight distinct types of noise, including ambiguous data types, empty functions, and syntax errors.
One particularly insidious type of noise is "No Relevance," where a test case exists in the dataset but does not actually exercise the logic of the focal method it is paired with. As shown in the example below, the test case calls a messaging API rather than the focal method it was supposed to test.
To validate these findings, the authors interviewed 17 domain experts, confirming that the identified noise categories are both prevalent and detrimental to model training.
The CleanTest Framework
The authors proposed CleanTest, an automated pipeline designed to scrub datasets of these identified noises. CleanTest utilizes three primary filters:
- Syntactic Filter: Uses Abstract Syntax Trees (ASTs) to identify and remove syntax errors, empty functions, and unnecessary annotations.
- Relevance Filter: A rule-based tool that ensures the test case actually invokes the focal method with appropriate parameters.
- Model-Based Coverage Filter: A fine-tuned CodeGPT model that predicts the branch coverage of a test. Tests that fail to exercise significant logic paths are discarded.
Experimental Results
The study evaluated CleanTest on two major datasets: Methods2Test and Atlas. The results revealed that noise is rampant, accounting for 43.52% of Methods2Test and 29.65% of Atlas.
By training four different LLMs (CodeBERT, AthenaTest, StarCoder, and CodeLlama7B) on the cleaned data, the researchers observed significant performance gains:
- Branch Coverage: Improved by an average of 67% for Methods2Test and 39% for Atlas.
- Bug Detection: Tests generated from clean data were 21.42% more effective at detecting real-world bugs in the Defects4J benchmark.
- Compilation Rate: For models like CodeBERT, the compilation rate of generated tests increased by over 500%.
Furthermore, because the training sets were smaller after filtering, training time was reduced by 10% to 25%, proving that "less is more" when data quality is prioritized.
Conclusion
This research highlights a critical bottleneck in AI-driven software engineering: the quality of training data. While current trends favor scaling dataset size, this study demonstrates that rigorous noise filtering can lead to better models with less compute. Future work will involve extending these language-specific filters to other programming languages and applying similar data-centric approaches to broader code generation tasks.
Summary updated