From Data Quality to Model Quality: an Exploratory Study on Deep Learning

Tianxing He, Shengcheng Yu, Ziyuan Wang, Jieqiong Li, Zhenyu Chen

Abstract

Nowadays, people strive to improve the accuracy of deep learning models. However, very little work has focused on the quality of data sets. In fact, data quality determines model quality. Therefore, it is important for us to make research on how data quality affects on model quality. In this paper, we mainly consider four aspects of data quality, including Dataset Equilibrium, Dataset Size, Quality of Label, and Dataset Contamination. We design experiments on MNIST and CIFAR-10 to find out the influence these four aspects have on model quality. Experimental results show that all four aspects have a decisive impact on the quality of models, meaning that a decrease in data quality in these aspects will reduce the accuracy of the model.

Introduction

In the field of deep learning, research often prioritizes architectural innovations—adding layers, refining attention mechanisms, or optimizing hyperparameters. However, the foundational principle of "garbage in, garbage out" remains paramount. While significant resources are devoted to model quality, the quality of the underlying datasets is frequently taken for granted. This study systematically quantifies how data degradation impacts model performance, focusing on class balance, dataset size, label accuracy, and image noise.

Deep learning success relies on a tripod of algorithms, hardware, and data. While the first two have seen explosive growth, data is often treated as a black box. This paper addresses this gap by investigating the sensitivity of models to specific types of data degradation, seeking to identify the points at which models fail to generalize.

Methodology

The researchers employed a "stress-test" methodology, intentionally introducing flaws into standard datasets to observe the resulting performance degradation.

Controlled Degradation

Clean datasets were modified by deleting classes, reducing sample sizes, flipping labels to incorrect values, and adding digital noise (e.g., Gaussian and Salt-and-Pepper noise).

Training to Memorization

Following the observation that deep networks can eventually memorize even random data, models were trained until they reached 100% training accuracy. This allowed researchers to evaluate how well the resulting "knowledge" generalized to a clean test set.

Datasets and Models

The study utilized two foundational benchmarks: MNIST (70,000 images of handwritten digits) and CIFAR-10 (60,000 color images across 10 categories). Models of varying complexity were tested, including LeNet-5, ResNet-20, and NetworkInNetwork (NIN).

Experimental Results

Dataset Size and Equilibrium

For CIFAR-10, the researchers identified a critical threshold for dataset size. Below 10,000 images, accuracy drops sharply. Above this point, while more data continues to improve performance, the rate of improvement slows significantly, suggesting a saturation point for dataset size relative to model complexity.

Impact of dataset size on model accuracy. Performance improves significantly at low percentages but reaches a saturation point as the dataset grows.

Furthermore, deleting an entire class does not merely affect the model's ability to recognize that specific class; it shifts decision boundaries for all classes, though the impact varies depending on the distinctiveness of the removed class.

Label Quality and Noise

The most critical finding is the fragility of models to label errors. While models can tolerate minor label noise, a "breaking point" exists where the model stops learning generalizable features and begins memorizing noise. On CIFAR-10, test accuracy plummeted toward random guessing (10%) as the label error rate approached 15-20%.

Model accuracy degradation as label noise increases. A sharp decline in performance is observed beyond a critical threshold of label corruption.

Dataset Contamination

Not all noise is detrimental. Small amounts of Gaussian noise can act as regularization, occasionally improving test accuracy. Conversely, Salt-and-Pepper noise and extreme contrast changes were found to be highly destructive. For instance, strengthening contrast on MNIST led to a drop in accuracy from 98% to 67%, as the model overfit to high-intensity features rather than digit shapes.

Conclusion

This exploratory study demonstrates that data quality is a decisive factor in model performance. The identification of a 20% label error threshold and the diminishing returns of dataset scale provide a quantitative foundation for data-centric AI. Future work will focus on developing a universal distance function to automatically score dataset quality and refining data augmentation strategies based on these insights into beneficial versus harmful noise types.

Summary updated

Abstract

Nowadays, people strive to improve the accuracy of deep learning models. However, very little work has focused on the quality of data sets. In fact, data quality determines model quality. Therefore, it is important for us to make research on how data quality affects on model quality. In this paper, we mainly consider four aspects of data quality, including Dataset Equilibrium, Dataset Size, Quality of Label, Dataset Contamination. We deign experiment on MNIST and Cifar-10 and try to find out the influence the four aspects make on model quality. Experimental results show that four aspects all have decisive impact on the quality of models. It means that decrease in data quality in these aspects will reduce the accuracy of model.
BibTeX
@inproceedings{he2019from,
  author = {Tianxing He and Shengcheng Yu and Ziyuan Wang and Jieqiong Li and Zhenyu Chen},
  title = {{From Data Quality to Model Quality: an Exploratory Study on Deep Learning}},
  booktitle = {Internetware},
  year = {2019},
  doi = {10.1145/3361242.3361260},
}
Figure