Learning Quality Labels for Robust Image Classification

Xiaosong Wang, Ziyue Xu, Dong Yang, Leo K. Tam, Holger Roth, Daguang Xu

Abstract

Training robust image classifiers often relies on high-quality annotations, which are frequently unavailable in large-scale or specialized domains like medical imaging. This paper presents a "learning-to-vote" meta-learning framework designed to handle datasets with multiple, conflicting, or noisy labels. By dynamically weighting different annotators during training through an attention-based meta-optimization process, the model learns to prioritize reliable information over noise. Experimental results on synthetic CIFAR-10 and real-world medical datasets (MIMIC-CXR) demonstrate that this approach significantly outperforms standard aggregation methods like majority voting, particularly in high-noise environments.

Introduction

In supervised learning, the assumption of a singular "ground truth" label often fails in practice. Medical imaging, for instance, is characterized by significant inter-observer variability among experts. Furthermore, large-scale datasets often rely on labels "mined" from unstructured text reports via natural language processing (NLP), introducing systematic errors. While existing robust learning methods typically address single noisy labels, modern datasets frequently provide multiple annotations from diverse sources. The core challenge lies in determining which labels to trust on a per-instance basis. Simple strategies like majority voting or averaging often fail to capture nuance or allow poor-quality annotators to degrade model performance.

Methods

The proposed framework introduces a Learning-to-Vote module that treats label selection as an integral part of the optimization process. Unlike static label aggregation, this meta-learning strategy evaluates the utility of each labeler during training.

The Learning-to-vote Module architecture, illustrating the meta-training process and the attention mechanism used to weight multiple annotations.

The training loop for each iteration consists of four primary stages:

  1. Meta-Training: For a given image, the model performs a trial gradient update for each available label set (e.g., $\mathbf{y}_1, \dots, \mathbf{y}_M$). This step assesses how the model's internal representations would shift if a specific labeler were believed.
  2. Attention Mechanism: Features are extracted from these trial updates. An attention module, utilizing a softmax function, assigns weights ($w_m$) to each labeler. High-quality labels that lead to more representative features receive higher weights.
  3. Label Refinement: The individual labels are aggregated into a weighted average $\bar{y}$. To maintain differentiability while ensuring "sharp" label signals, a differentiable binarization function is applied to produce the refined label $\tilde{y}$.
  4. Final Parameter Update: The model's parameters are updated using the refined label. Because gradients flow through the entire module, the system effectively learns to optimize the weighting mechanism itself to minimize final loss.

Experimental Setup

The framework was validated across diverse scenarios:

  • Synthetic CIFAR-10: To test robustness, researchers simulated five distinct types of "bad" annotators, including random "Hammer-Spammers" and "Adversarial" labelers who provide intentionally incorrect labels.
  • MIMIC-CXR: A large-scale chest X-ray dataset where labels were automatically extracted using two different NLP tools (Negbio and Chexpert).
  • Real-world Benchmarks: Performance was further verified on a 1,000-image subset of MIMIC-CXR annotated by a board-certified radiologist and the independent OpenI public dataset.

Results and Discussion

The "Learning-to-Vote" approach consistently surpassed baseline methods, including simple averaging (AVG), majority voting (MV), and previous state-of-the-art confusion matrix (CM) approaches.

Performance comparison of the proposed method against baselines (AVG, MV, CM) across different noise levels on the CIFAR-10 dataset.

On the synthetic CIFAR-10 task, the model achieved 70.5% accuracy in high-noise scenarios, significantly higher than MV (61.1%) and CM (64.3%). In medical imaging tasks on the MIMIC-CXR dataset, the method yielded a 3-5% average increase in Area Under the Curve (AUC) across 14 pathologies.

A key insight is the model's ability to handle systematic error; in adversarial tests, it learned to assign near-zero weights to intentionally misleading annotators. Furthermore, the framework demonstrated strong cross-dataset generalization, maintaining superior performance when trained on noisy MIMIC data and tested on the clean OpenI dataset. When integrated with multimodal features (e.g., BioBERT embeddings), the system often produced labels more accurate than the NLP-mined sources used for its own training.

Conclusion

The "Learning-to-Vote" framework provides a robust solution for training on datasets with multiple, noisy annotations. By leveraging meta-learning to dynamically weight labelers, it moves beyond the limitations of simple majority consensus. While the approach introduces additional computational overhead during training due to multiple back-propagations, the gains in accuracy and robustness—especially in sensitive domains like medical diagnostics—are substantial. Future work may explore more complex self-attention mechanisms and strategies to address class imbalance in long-tailed datasets.

Summary updated

Abstract

Supervised learning paradigms largely benefit from the tremendous amount of annotated data. However, the quality of the annotations often varies among labelers. Multi-observer studies have been conducted to examine the annotation variances (by labeling the same data multiple times) to see how it affects critical applications like medical image analysis. In this paper, we demonstrate how multiple sets of annotations (either hand-labeled or algorithm-generated) can be utilized together and mutually benefit the learning of classification tasks. A scheme of learning-to-vote is introduced to sample quality label sets for each data entry on-the-fly during the training. Specifically, a label-sampling module is designed to achieve refined labels (weighted sum of attended ones) that benefit the model learning the most through additional back-propagations. We apply the learning-to-vote scheme on the classification task of a synthetic noisy CIFAR-10 to prove the concept and then demonstrate superior results (3-5% increase on average in multiple disease classification AUCs) on the chest x-ray images from a hospital-scale dataset (MIMIC-CXR) and hand-labeled dataset (OpenI) in comparison to regular training paradigms.
BibTeX
@inproceedings{wang2024learning,
  author = {Xiaosong Wang and Ziyue Xu and Dong Yang and Leo K. Tam and Holger Roth and Daguang Xu},
  title = {{Learning Quality Labels for Robust Image Classification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00114},
}
Figure