Simplification of Training Data for Cross-Project Defect Prediction

Peng He, Bing Li, Deguang Zhang, Yutao Ma

Abstract

Cross-project defect prediction (CPDP) plays an important role in estimating the most likely defect-prone software components, especially for new or inactive projects. To the best of our knowledge, few prior studies provide explicit guidelines on how to select suitable training data of quality from a large number of public software repositories. In this paper, we have proposed a training data simplification method for practical CPDP in consideration of multiple levels of granularity and filtering strategies for data sets. In addition, we have also provided quantitative evidence on the selection of a suitable filter in terms of defect-proneness ratio. Based on an empirical study on 34 releases of 10 open-source projects, we have elaborately compared the prediction performance of different defect predictors built with five well-known classifiers using training data simplified at different levels of granularity and with two popular filters. The results indicate that when using the multi-granularity simplification method with an appropriate filter, the prediction models based on Naive Bayes can achieve fairly good performance and outperform the benchmark method.

The Challenge of Cross-Project Defect Prediction

Software defect prediction is a critical task for identifying "defect-prone" components, allowing engineers to focus testing resources where they are most needed. Traditionally, this relies on Within-Project Defect Prediction (WPDP), which uses a project's own historical data. However, new or inactive projects often lack the necessary history to train effective models.

Cross-Project Defect Prediction (CPDP) addresses this by borrowing training data from external open-source repositories. The primary challenge is that external data often introduces noise; differences in coding style, complexity, and domain between projects can lead to high false-alarm rates. The core problem is one of data quality: how to select only the most relevant training examples from a massive pool of external data.

The riTDS Methodology

To address the data quality issue, we propose a multi-granularity approach called riTDS (simplified Training Data Selection). This method operates in two distinct stages to prune irrelevant data effectively.

Overview of the defect prediction and data simplification framework.

1. Coarse-grained (Release-level) Selection

Instead of considering all external projects, the first stage identifies the $k$ most similar software releases. This similarity is determined by comparing high-level statistical distributions—such as the mean, median, maximum, minimum, and standard deviation—of software metrics across different releases.

The process of aggregating instance-level metrics into a release-level feature vector for similarity computation.

2. Fine-grained (Instance-level) Selection

Once similar releases are identified, the data is further refined using one of two filtering strategies:

  • riTDS-1 (Test set-driven): This filter selects the nearest neighbors for each instance in the target project, ensuring the training data closely matches the specific characteristics of the project being predicted.
  • riTDS-2 (Training set-driven): A more restrictive filter that labels training instances based on their proximity to target instances, further reducing the volume of training data.

To guide practitioners, we introduce the Defect Proneness Ratio (DPR)—the ratio of the defect proportion in the training set to that in the test set. This serves as a quantitative rule for choosing the most appropriate filter for a given scenario.

Experimental Setup and Results

Our study utilized 34 releases from 10 well-known open-source projects (e.g., Apache Ant, Camel, Lucene, and Xalan) from the PROMISE repository. Each dataset included 20 software metrics, such as lines of code and coupling depth.

Key Findings

  • Quality Over Quantity: Massive training sets are often a liability. Our simplified sets were sometimes 50 times smaller than the original pool but yielded more accurate predictions. The average training size was reduced from 11,824 instances to as few as 218.
  • Classifier Performance: Among five tested classifiers (J48, Logistic Regression, Naive Bayes, Random Forest, and SVM), Naive Bayes consistently performed best. Its simplicity makes it robust against the distribution shifts common in cross-project data, achieving an average f-measure of 0.475.
  • Hybrid Advantage: Filtering at both the release and instance levels is significantly more effective than using either approach in isolation.

Conclusion and Future Work

The riTDS method demonstrates that "less is more" in CPDP. By focusing on data quality through multi-granularity simplification, we can achieve satisfactory performance with significantly lower computational overhead. Future work will involve expanding this study to proprietary software and different programming languages, as well as moving beyond binary labels to predict the specific number of defects in a component.

Summary updated

Abstract

Cross-project defect prediction (CPDP) plays an important role in estimating the most likely defect-prone software components, especially for new or inactive projects. To the best of our knowledge, few prior studies provide explicit guidelines on how to select suitable training data of quality from a large number of public software repositories. In this paper, we have proposed a training data simplification method for practical CPDP in consideration of multiple levels of granularity and filtering strategies for data sets. In addition, we have also provided quantitative evidence on the selection of a suitable filter in terms of defect-proneness ratio. Based on an empirical study on 34 releases of 10 open-source projects, we have elaborately compared the prediction performance of different defect predictors built with five well-known classifiers using training data simplified at different levels of granularity and with two popular filters. The results indicate that when using the multi-granularity simplification method with an appropriate filter, the prediction models based on Naive Bayes can achieve fairly good performance and outperform the benchmark method.
BibTeX
@misc{he2014simplification,
  author = {Peng He and Bing Li and Deguang Zhang and Yutao Ma},
  title = {{Simplification of Training Data for Cross-Project Defect Prediction}},
  year = {2014},
  eprint = {1405.0773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1405.0773},
}
Figure