Do Deep Neural Networks Always Perform Better When Eating More Data?

Jiachen Yang, Zhuo Zhang, Yicheng Gong, Shukun Ma, Xiaolan Guo, Yue Yang, Shuai Xiao, Jiabao Wen, Yang Li, Xinbo Gao, Wen Lu, Qinggang MengarXiv 2022 · DOI · arXiv

Abstract

This paper challenges the prevailing "more is always better" dogma in deep learning by demonstrating that data quality and information density often trump raw volume. By introducing a "two-tier amount of information" theory and investigating "bias-fitting" in out-of-distribution (OOD) scenarios, we show that smaller, carefully curated datasets can significantly outperform much larger, poorly selected ones. Our findings suggest that prioritizing high-information samples can lead to equivalent performance with less than half the data, offering a more efficient path for data-centric AI development.

Introduction

The prevailing trend in deep learning is to "feed the beast"—blindly expanding datasets to improve model accuracy. However, this approach is expensive, computationally wasteful, and often yields diminishing returns. A critical gap in current research is the lack of a convincing guiding theory to determine which data actually helps a model learn. This study addresses this gap by examining two specific scenarios: Independent and Identically Distributed (IID) settings and Out-of-Distribution (OOD) settings.

Methodology

The researchers employed several Information Evaluation Indicators (IEIs) to rank the "contribution" of individual data samples, including Entropy (measuring output uncertainty), Distance Entropy (mapping samples to feature space relative to class prototypes), and Metric (assessing intra-class and inter-class relationships).

Overview of the IID and OOD experimental frameworks for data information evaluation.

As illustrated in the framework above, the study utilized these IEIs to conduct "Addition" and "Reduction" experiments. In addition experiments, a small base set was supplemented with either a "Goodset" (high information) or a "Badset" (low information). Reduction experiments involved removing samples to observe performance degradation. For OOD analysis, Euclidean distance was used to separate "positive migration data" (samples similar to the test domain) from "negative migration data" (samples that mislead the network).

Experiments and Datasets

The study utilized four key benchmarks:

  • IID Benchmarks: The classic CIFAR-10 and Mini-ImageNet were used to test information density theories.
  • OOD Benchmarks:
    • NICO-Animal: A dataset featuring animals in various contexts (e.g., a dog in water vs. a dog in a cage).
  • CSE (Full Elements Dataset): A custom dataset captured using a six-degree-of-freedom platform to control angles, distances, and backgrounds precisely.

Key Insights and Ideas

  • The Two-Tier Information Theory: Information exists at two levels: Amount of Class Information (ACI), where some classes are inherently harder to learn, and Amount of Sample Information (ASI), where specific samples within a class provide higher unique value.
  • Negative Migration: In OOD settings, adding more data can be counterproductive. If new data is "far" from the target domain, it introduces noise that reduces performance.
  • Bias-Fitting: Models often take "shortcuts." If training images of a dog always include a cage, the model learns to identify the cage rather than the dog. This "bias-fitting" causes failure when the model is tested on dogs in the wild.

Key Results

The experiments revealed significant efficiency gains when prioritizing high-information data. On CIFAR-10, a model trained on just 30% High Information Density (HID) data achieved the same accuracy as one trained on 73% Low Information Density (LID) data, representing a saving of roughly 21,500 samples.

Accuracy vs. Sample Ratio for various Information Evaluation Indicators (IEIs).

As shown in the performance curves, the "Goodset" consistently maintains higher accuracy with fewer samples compared to the "Badset." In OOD experiments, a split containing only 40% "positive migration" samples consistently outperformed a split containing 60% "negative migration" samples. Furthermore, the authors found that "unbalanced" budget schemes—allocating more samples to high-ACI (hard) classes—is significantly more efficient than uniform allocation.

Conclusion

This research provides a strong framework for data-centric AI, proving that data quality is a more reliable driver of performance than raw quantity. By focusing on information density and mitigating bias-fitting, models can be trained more efficiently. Future work will aim to expand these theories to complex tasks like object detection and develop more rigorous mathematical models for sample information.

Summary updated

Abstract

Data has now become a shortcoming of deep learning. Researchers in their own fields share the thinking that "deep neural networks might not always perform better when they eat more data," which still lacks experimental validation and a convincing guiding theory. Here to fill this lack, we design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD), which give powerful answers. For the purpose of guidance, based on the discussion of results, two theories are proposed: under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of sample information and the amount of class information; under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain. The above theories provide guidance from the perspective of data, which can promote a wide range of practical applications of artificial intelligence.
BibTeX
@misc{yang2022do,
  author = {Jiachen Yang and Zhuo Zhang and Yicheng Gong and Shukun Ma and Xiaolan Guo and Yue Yang and Shuai Xiao and Jiabao Wen and Yang Li and Xinbo Gao and Wen Lu and Qinggang Meng},
  title = {{Do Deep Neural Networks Always Perform Better When Eating More Data?}},
  year = {2022},
  eprint = {2205.15187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.15187},
}
Figure