An Empirical Study of the Noise Impact on Cost-Sensitive Learning

Xingquan Zhu, Xindong Wu, Taghi M. Khoshgoftaar, Yong Shi

Abstract

Cost-sensitive (CS) learning is vital for applications where misclassification costs are asymmetric, such as medical diagnosis or fraud detection. This study investigates the robustness of CS models—specifically decision trees—against mislabeled training data. Our findings indicate that CS models are significantly more fragile than standard accuracy-based models, particularly as the cost-ratio increases. We identify a "saturation point" where models fail to learn effectively from noisy data and demonstrate that standard data cleansing techniques are essential for maintaining performance in cost-sensitive contexts.

Introduction

In many real-world scenarios, the cost of a False Negative (e.g., missing a disease) far outweighs the cost of a False Positive. Cost-sensitive (CS) learning addresses this by optimizing for total financial or physical cost rather than simple error rates. However, training data is often corrupted by noise. While the impact of noise on standard classifiers is well-documented, its effect on CS learners remains under-explored. This paper examines how mislabeled data affects models trying to minimize costs and evaluates the role of the cost-ratio ($r$) in this sensitivity.

Methodology

The study employs the C5.0 classification tree as the base learner. To isolate the effects of noise, two corruption mechanisms were used: Total Random Corruption (TRC) and Proportional Random Corruption (PRC). The latter ensures that the class distribution remains constant, allowing researchers to attribute performance changes solely to label noise. The experiments varied the cost-ratio ($r$)—the relative penalty for misclassifying the minority class—across three UCI benchmark datasets: Credit Screening, WDBC, and Thyroid Disease.

The Fragility of Cost-Sensitive Learning

A central discovery of this research is the "saturation point." As noise levels increase, CS learners initially adjust their decision boundaries. However, beyond a certain threshold, the model becomes insensitive to additional noise, essentially "giving up" on accuracy to minimize costs by predicting the cheapest class.

Average costs vs. noise level showing the saturation point and the effectiveness of data cleansing (CS_E'').

As shown in Figure 1, the average cost for models trained on noisy data ($CS\_E'$) rises sharply with noise, especially for high cost-ratios ($r=10$). In contrast, models trained on cleansed data ($CS\_E''$) maintain stable, low costs, highlighting the critical importance of data preprocessing.

Empirical Results and Data Cleansing

The sensitivity of CS models is directly proportional to the cost-ratio. In the WDBC dataset, a cost-ratio of $r=10$ caused a dramatic increase in average misclassification costs compared to lower ratios.

Detailed view of cost inflation on the WDBC dataset, highlighting the sensitivity to high cost-ratios (r=10).

Figure 6 illustrates this trend, where the $r=10$ curve (marked with asterisks) exhibits the most significant cost inflation as noise increases. The study also theoretically proves that CS learners are inherently less accurate than standard classifiers because they intentionally sacrifice accuracy on "cheap" instances. Consequently, the authors recommend using standard, accuracy-focused classifiers for the initial data cleansing phase to identify and remove noisy labels effectively.

Conclusion

This empirical study highlights a significant vulnerability in cost-sensitive learning: extreme sensitivity to label noise. As the stakes (cost-ratios) rise, so does the fragility of the model. Future work should extend these findings to other architectures like SVMs and Neural Networks and explore more complex multi-class cost matrices. For practitioners, the message is clear: data cleansing is not an optional step but a fundamental requirement for reliable cost-sensitive applications.

Summary updated

Abstract

In this paper, we perform an empirical study of the impact of noise on cost-sensitive (CS) learning, through observations on how a CS learner reacts to the mislabeled training examples in terms of misclassification cost and classification accuracy. Our empirical results and theoretical analysis indicate that mislabeled training examples can raise serious concerns for cost-sensitive classification, especially when misclassifying some classes becomes extremely expensive. Compared to general inductive learning, the problem of noise handling and data cleansing is more crucial, and should be carefully investigated to ensure the success of CS learning.
BibTeX
@inproceedings{zhu2007empirical,
  author = {Xingquan Zhu and Xindong Wu and Taghi M. Khoshgoftaar and Yong Shi},
  title = {{An Empirical Study of the Noise Impact on Cost-Sensitive Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
}
Figure