Less Is More: On the Importance of Data Quality for Unit Test Generation

Junwei Zhang, Xing Hu, Shan Gao, Xin Xia, David Lo, Shanping Li

In the world of Large Language Models (LLMs), the mantra has long been "more is better." We scrape billions of tokens, assuming the model will eventually separate the wheat from the chaff. However, a new paper from researchers at Zhejiang University, Huawei, and SMU suggests that when it comes to automated unit test generation, we’ve been feeding our models a lot of "chaff."

TL;DR

Researchers discovered that up to 43% of the data in popular unit test generation datasets is "noise"—ranging from syntax errors to tests that don't actually exercise the code they claim to. By introducing CleanTest, a framework that filters this noise, they improved model performance (branch coverage) by up to 67% while actually reducing training time.

What Problem Does This Paper Address?

Automated unit test generation is a holy grail for software engineers. If an AI can write high-quality tests, developers can focus on building features rather than chasing edge cases. Current state-of-the-art approaches fine-tune LLMs on massive datasets like Methods2Test or Atlas.

The problem? These datasets are often mined from GitHub with minimal filtering. If a model is trained on "noisy" data—tests that are syntactically broken, irrelevant to the code under test, or provide zero coverage—the model learns to generate equally poor tests. This paper is the first to systematically categorize this noise and measure its impact on model performance.

Methods

The authors approached the problem in three stages:

  1. Taxonomy Construction: Using a method called "open card sorting," researchers manually inspected 50,000 pairs of focal methods (the code being tested) and their corresponding test cases. They identified eight distinct types of noise, such as "Ambiguous Data Types" and "No Relevance."
  2. Expert Validation: They interviewed 17 domain experts to ensure their noise categories were reasonable and comprehensive.
  3. The CleanTest Framework: They built an automated pipeline to scrub datasets using three filters:
  4. Syntactic Filter: A rule-based tool that uses Abstract Syntax Trees (ASTs) to catch syntax errors, empty functions, and unnecessary annotations.
  5. Relevance Filter: This ensures the test case actually calls the "focal method" with the correct parameters.
  6. Model-Based Coverage Filter: They fine-tuned a version of CodeGPT to predict the "branch coverage" of a test. If a test doesn't exercise enough logic paths, it's discarded.

Datasets

The study utilized two primary training datasets and one gold-standard evaluation benchmark:

  • Methods2Test: Currently the largest and most popular dataset for this task (mined from GitHub).
  • Atlas: A widely used dataset where each test case contains only a single assertion.
  • Defects4J (Evaluation): A famous benchmark containing real-world bugs from 17 Java open-source systems. This was used to see if the generated tests could actually find real bugs.

Key Insights and Ideas

  • Noise is Everywhere: The researchers found that 43.52% of the Methods2Test dataset and 29.65% of the Atlas dataset consisted of noise. This is a staggering amount of "bad data" for models to be learning from.
  • The "Less is More" Paradox: By removing nearly half the data, the models didn't get worse—they got significantly better. This challenges the "scale at all costs" mentality in AI training.
  • Syntactic Correctness $\neq$ Quality: A test can be perfectly valid Java code but still be "noise" if it doesn't actually test the logic of the target function. The Relevance Filter was crucial here, as it caught cases where a test was simply testing a different method with a similar name.
  • Efficiency Gains: Because the training sets were smaller after filtering, the authors saved between 10% and 25% in training time while achieving superior results.

Key Results

The impact of using CleanTest was dramatic across four different LLMs (CodeBERT, AthenaTest, StarCoder, and CodeLlama7B):

  • Branch Coverage: Fine-tuning on filtered data improved branch coverage by an average of 67% for Methods2Test and 39% for Atlas.
  • Bug Detection: Tests generated by models trained on clean data were 21.42% more effective at detecting real-world bugs in the Defects4J benchmark.
  • Compilation Rate: For CodeBERT, the number of generated tests that actually compiled successfully jumped by a massive 578% when trained on clean data.

Limitations and Future Work

While the results are impressive, the authors acknowledge a few hurdles:

  • Language Specificity: The current heuristic rules and AST-based filters are tailored for Java. Expanding this to Python, C++, or Go would require developing new language-specific rules.
  • The "Coverage" Proxy: The model-based coverage filter relies on a predicted metric. While effective, it’s still an approximation of real-world execution.
  • Beyond Unit Tests: The authors suggest that this "data-centric" approach should be applied to other software engineering tasks, such as automated code review and general code generation, where noise is likely just as prevalent.

In short: If you want your AI to write better code, stop giving it more data and start giving it better data.

Summary updated

Abstract

Unit testing is crucial for software development and maintenance. Effective unit testing ensures and improves software quality, but writing unit tests is time-consuming and labor-intensive. Recent studies have proposed deep learning (DL) techniques or large language models (LLMs) to automate unit test generation. These models are usually trained or fine-tuned on large-scale datasets. Despite growing awareness of the importance of data quality, there has been limited research on the quality of datasets used for test generation. To bridge this gap, we systematically examine the impact of noise on the performance of learning-based test generation models. We first apply the open card sorting method to analyze the most popular and largest test generation dataset, Methods2Test, to categorize eight distinct types of noise. Further, we conduct detailed interviews with 17 domain experts to validate and assess the importance, reasonableness, and correctness of the noise taxonomy. Then, we propose CleanTest, an automated noise-cleaning framework designed to improve the quality of test generation datasets. CleanTest comprises three filters: a rule-based syntax filter, a rule-based relevance filter, and a model-based coverage filter. To evaluate its effectiveness, we apply CleanTest on two widely-used test generation datasets, i.e., Methods2Test and Atlas. Our findings indicate that 43.52% and 29.65% of datasets contain noise, highlighting its prevalence. Finally, we conduct comparative experiments using four LLMs (i.e., CodeBERT, AthenaTest, StarCoder, and CodeLlama7B) to assess the impact of noise on test generation performance. The results show that filtering noise positively influences the test generation ability of the models.
BibTeX
@article{zhang2025less,
  author = {Junwei Zhang and Xing Hu and Shan Gao and Xin Xia and David Lo and Shanping Li},
  title = {{Less Is More: On the Importance of Data Quality for Unit Test Generation}},
  journal = {Proc. ACM Softw. Eng.},
  year = {2025},
  doi = {10.1145/3715778},
}

Loading related content...

Figure