In traditional machine learning, we often treat "ground truth" as an absolute certainty. We assume an image of a cat is 100% "cat." However, human annotators frequently disagree due to simple mistakes (noise) or inherent visual ambiguity (e.g., is that a bobcat or a house cat?).
Most research takes a model-centric approach, holding the data constant while searching for the best architecture. This paper argues for a data-centric shift. The authors point out that aggregating multiple human opinions into a single "majority vote" label actually discards valuable information about aleatoric uncertainty—the inherent randomness or ambiguity in the data itself.
To address this gap, the authors introduce the Data-Centric Image Classification (DCIC) benchmark. This framework is designed to measure how well algorithms can recover the "true" underlying distribution of labels from noisy, multi-annotator input, proving that better label estimation drives performance just as much as better model architecture.
The DCIC Benchmark: Methods and Datasets
The authors propose a two-phase benchmark structure to isolate the impact of label quality:
- Labeling Phase: A "teacher" method (the algorithm being benchmarked) takes a limited budget of human annotations and attempts to produce the best possible labels for the training set.
- Evaluation Phase: To ensure a fair comparison, a fixed student model is trained on these improved labels. Its performance is then measured on a clean test set.
By fixing the model in Phase 2, the benchmark strictly evaluates the quality of the labels. The authors implemented 20 different baseline methods—including supervised, semi-supervised (like FixMatch), and self-supervised (like SimCLR) approaches—to see which could best "clean" or estimate the true labels.
The DCIC benchmark consists of ten datasets across diverse domains, many of which were specifically extended with additional human annotations for this study:
- Biological/Environmental: Benthic (seafloor images), Plankton, MiceBone (collagen fibers), Turkey (injury detection), and Pig (tail lesions).
- Medical: Quality MRI (subjective quality ratings).
- Standard/Synthetic: CIFAR-10H (the classic dataset with 50 human labels per image), Synthetic (shapes with controlled ambiguity), and two versions of TreeVersity (crowdsourced plant images).
Key Insights and Results
The benchmark reveals several critical insights about how we handle data in machine learning:
- The "One Annotation" Fallacy: The authors empirically prove that one label is rarely enough. On average, moving from a single annotation to a full set of annotations increased accuracy from roughly 55% to 72% across the benchmarked tasks. Model performance scales significantly as you add more annotations per image, especially for ambiguous samples.
- Hard Labels vs. Soft Labels: Forcing a model to learn "hard" (one-hot) labels for ambiguous images leads to overconfident models. Using "soft" labels (probability distributions) better captures the subjective disagreement of humans. This significantly reduced the Expected Calibration Error (ECE), meaning the models became much better at "knowing what they don't know."
- The SOTA Struggle: Surprisingly, many state-of-the-art (SOTA) noise-reduction algorithms actually perform worse than a simple supervised baseline when the annotation budget is high. This suggests that current "noise-cleansing" AI is better at fixing random mistakes than handling nuanced, human-driven ambiguity.
- Human Performance Gap: In several datasets (like Benthic and Pig), the baseline model's accuracy was over 10% lower than the "human agreement" rate, highlighting these as particularly challenging data-centric problems.
Limitations and Future Work
The authors acknowledge a few constraints in their current study. The datasets are relatively small (all under 15,000 images), making it difficult to test how these methods scale to massive datasets. Additionally, to save on computational costs (which still totaled 5,500 GPU hours), they used fixed hyperparameters across datasets rather than tuning for each specific one. Finally, the benchmark currently uses a fixed initialization and does not yet explore active learning, where the model actively chooses which images need more annotations.
Ultimately, the paper concludes that the machine learning community needs more robust algorithms specifically designed to handle subjective ambiguity rather than just random label noise. To encourage further research, the authors have released their source code and datasets to the public.
Summary updated