Abstract
Training robust image classifiers often relies on high-quality annotations, which are frequently unavailable in large-scale or specialized domains like medical imaging. This paper presents a "learning-to-vote" meta-learning framework designed to handle datasets with multiple, conflicting, or noisy labels. By dynamically weighting different annotators during training through an attention-based meta-optimization process, the model learns to prioritize reliable information over noise. Experimental results on synthetic CIFAR-10 and real-world medical datasets (MIMIC-CXR) demonstrate that this approach significantly outperforms standard aggregation methods like majority voting, particularly in high-noise environments.
Introduction
In supervised learning, the assumption of a singular "ground truth" label often fails in practice. Medical imaging, for instance, is characterized by significant inter-observer variability among experts. Furthermore, large-scale datasets often rely on labels "mined" from unstructured text reports via natural language processing (NLP), introducing systematic errors. While existing robust learning methods typically address single noisy labels, modern datasets frequently provide multiple annotations from diverse sources. The core challenge lies in determining which labels to trust on a per-instance basis. Simple strategies like majority voting or averaging often fail to capture nuance or allow poor-quality annotators to degrade model performance.
Methods
The proposed framework introduces a Learning-to-Vote module that treats label selection as an integral part of the optimization process. Unlike static label aggregation, this meta-learning strategy evaluates the utility of each labeler during training.
The training loop for each iteration consists of four primary stages:
- Meta-Training: For a given image, the model performs a trial gradient update for each available label set (e.g., $\mathbf{y}_1, \dots, \mathbf{y}_M$). This step assesses how the model's internal representations would shift if a specific labeler were believed.
- Attention Mechanism: Features are extracted from these trial updates. An attention module, utilizing a softmax function, assigns weights ($w_m$) to each labeler. High-quality labels that lead to more representative features receive higher weights.
- Label Refinement: The individual labels are aggregated into a weighted average $\bar{y}$. To maintain differentiability while ensuring "sharp" label signals, a differentiable binarization function is applied to produce the refined label $\tilde{y}$.
- Final Parameter Update: The model's parameters are updated using the refined label. Because gradients flow through the entire module, the system effectively learns to optimize the weighting mechanism itself to minimize final loss.
Experimental Setup
The framework was validated across diverse scenarios:
- Synthetic CIFAR-10: To test robustness, researchers simulated five distinct types of "bad" annotators, including random "Hammer-Spammers" and "Adversarial" labelers who provide intentionally incorrect labels.
- MIMIC-CXR: A large-scale chest X-ray dataset where labels were automatically extracted using two different NLP tools (Negbio and Chexpert).
- Real-world Benchmarks: Performance was further verified on a 1,000-image subset of MIMIC-CXR annotated by a board-certified radiologist and the independent OpenI public dataset.
Results and Discussion
The "Learning-to-Vote" approach consistently surpassed baseline methods, including simple averaging (AVG), majority voting (MV), and previous state-of-the-art confusion matrix (CM) approaches.
On the synthetic CIFAR-10 task, the model achieved 70.5% accuracy in high-noise scenarios, significantly higher than MV (61.1%) and CM (64.3%). In medical imaging tasks on the MIMIC-CXR dataset, the method yielded a 3-5% average increase in Area Under the Curve (AUC) across 14 pathologies.
A key insight is the model's ability to handle systematic error; in adversarial tests, it learned to assign near-zero weights to intentionally misleading annotators. Furthermore, the framework demonstrated strong cross-dataset generalization, maintaining superior performance when trained on noisy MIMIC data and tested on the clean OpenI dataset. When integrated with multimodal features (e.g., BioBERT embeddings), the system often produced labels more accurate than the NLP-mined sources used for its own training.
Conclusion
The "Learning-to-Vote" framework provides a robust solution for training on datasets with multiple, noisy annotations. By leveraging meta-learning to dynamically weight labelers, it moves beyond the limitations of simple majority consensus. While the approach introduces additional computational overhead during training due to multiple back-propagations, the gains in accuracy and robustness—especially in sensitive domains like medical diagnostics—are substantial. Future work may explore more complex self-attention mechanisms and strategies to address class imbalance in long-tailed datasets.
Summary updated