Abstract
This paper challenges the prevailing "more is always better" dogma in deep learning by demonstrating that data quality and information density often trump raw volume. By introducing a "two-tier amount of information" theory and investigating "bias-fitting" in out-of-distribution (OOD) scenarios, we show that smaller, carefully curated datasets can significantly outperform much larger, poorly selected ones. Our findings suggest that prioritizing high-information samples can lead to equivalent performance with less than half the data, offering a more efficient path for data-centric AI development.
Introduction
The prevailing trend in deep learning is to "feed the beast"—blindly expanding datasets to improve model accuracy. However, this approach is expensive, computationally wasteful, and often yields diminishing returns. A critical gap in current research is the lack of a convincing guiding theory to determine which data actually helps a model learn. This study addresses this gap by examining two specific scenarios: Independent and Identically Distributed (IID) settings and Out-of-Distribution (OOD) settings.
Methodology
The researchers employed several Information Evaluation Indicators (IEIs) to rank the "contribution" of individual data samples, including Entropy (measuring output uncertainty), Distance Entropy (mapping samples to feature space relative to class prototypes), and Metric (assessing intra-class and inter-class relationships).
As illustrated in the framework above, the study utilized these IEIs to conduct "Addition" and "Reduction" experiments. In addition experiments, a small base set was supplemented with either a "Goodset" (high information) or a "Badset" (low information). Reduction experiments involved removing samples to observe performance degradation. For OOD analysis, Euclidean distance was used to separate "positive migration data" (samples similar to the test domain) from "negative migration data" (samples that mislead the network).
Experiments and Datasets
The study utilized four key benchmarks:
- IID Benchmarks: The classic CIFAR-10 and Mini-ImageNet were used to test information density theories.
- OOD Benchmarks:
- NICO-Animal: A dataset featuring animals in various contexts (e.g., a dog in water vs. a dog in a cage).
- CSE (Full Elements Dataset): A custom dataset captured using a six-degree-of-freedom platform to control angles, distances, and backgrounds precisely.
Key Insights and Ideas
- The Two-Tier Information Theory: Information exists at two levels: Amount of Class Information (ACI), where some classes are inherently harder to learn, and Amount of Sample Information (ASI), where specific samples within a class provide higher unique value.
- Negative Migration: In OOD settings, adding more data can be counterproductive. If new data is "far" from the target domain, it introduces noise that reduces performance.
- Bias-Fitting: Models often take "shortcuts." If training images of a dog always include a cage, the model learns to identify the cage rather than the dog. This "bias-fitting" causes failure when the model is tested on dogs in the wild.
Key Results
The experiments revealed significant efficiency gains when prioritizing high-information data. On CIFAR-10, a model trained on just 30% High Information Density (HID) data achieved the same accuracy as one trained on 73% Low Information Density (LID) data, representing a saving of roughly 21,500 samples.
As shown in the performance curves, the "Goodset" consistently maintains higher accuracy with fewer samples compared to the "Badset." In OOD experiments, a split containing only 40% "positive migration" samples consistently outperformed a split containing 60% "negative migration" samples. Furthermore, the authors found that "unbalanced" budget schemes—allocating more samples to high-ACI (hard) classes—is significantly more efficient than uniform allocation.
Conclusion
This research provides a strong framework for data-centric AI, proving that data quality is a more reliable driver of performance than raw quantity. By focusing on information density and mitigating bias-fitting, models can be trained more efficiently. Future work will aim to expand these theories to complex tasks like object detection and develop more rigorous mathematical models for sample information.
Summary updated