Abstract
Deep Neural Networks (DNNs) are highly susceptible to memorizing noisy labels in large-scale, web-scraped datasets, which severely compromises their generalization capabilities. This paper proposes Attentive Feature Mixup (AFM), a novel training module designed to identify and down-weight mislabeled samples dynamically. By grouping samples and applying a self-attention mechanism, AFM generates "noise-suppressed" synthetic samples that provide a cleaner training signal. AFM is an assumption-free, plug-and-play solution that achieves state-of-the-art results on real-world benchmarks like Food101N and Clothing1M without requiring auxiliary clean data.
Introduction
The success of DNNs is often predicated on the availability of massive, accurately labeled datasets. However, real-world data collection—such as scraping images based on tags—inevitably introduces label noise. Standard training procedures lead models to memorize these errors, destroying their ability to generalize to clean test data. While previous research has attempted to mitigate this through clean subsets or noise-modeling, these approaches are often limited by rigid assumptions or the need for expensive manual verification.
AFM addresses these challenges by treating noise detection as a relative problem: by comparing samples within small groups, the model can more easily identify "odd one out" instances that do not align with the group's dominant features.
Methodology: Attentive Feature Mixup (AFM)
The AFM block is integrated into the training pipeline and consists of two primary components: the Group-to-Attend (GA) module and the Mixup module.
Group-to-Attend (GA) Module
In each mini-batch, the GA module randomly selects a small group of $K$ samples (typically $K=2$). These samples are projected into an embedding space via separate fully-connected layers ($FC_a$ and $FC_b$). The embeddings interact—most effectively through summation—to allow the network to compare features. A lightweight self-attention network then assigns a weight ($\alpha$) to each sample. The objective is for the network to learn to assign higher weights to clean, representative samples and lower weights to those suspected of being mislabeled.
Mixup and Multi-task Learning
The Mixup module uses the computed attention weights to perform a weighted interpolation of the group's features and labels, creating a "noise-suppressed" virtual sample. The model is trained using a multi-task loss: one classifier ($f_{c2}$) processes the original noisy samples, while another ($f_{c1}$) processes the synthetic, cleaner samples. This dual-path approach ensures the model learns robust features from the mixed data while still seeing the full dataset.
Theoretical Motivation and Noise Dilution
The effectiveness of AFM stems from the principle of "noise dilution." While a dataset may have a high overall noise ratio, the probability of selecting a group where every member is mislabeled is significantly lower. By mixing samples, the influence of a single noisy label is diluted by the presence of clean data in the same group.
As shown in Figure 1, this process effectively expands the decision boundaries of the classes. The authors also demonstrate that for the attention weights to be discriminative, the projection layers for group members must be distinct ($f_a \neq f_b$), ensuring the network can correctly associate weights with specific samples.
Experimental Results
The authors evaluated AFM on two large-scale, real-world noisy datasets: Food101N (365k images, ~20% noise) and Clothing1M (1M images, ~38.5% noise).
- Food101N: AFM achieved a top-1 accuracy of 87.23%, outperforming the ResNet50 baseline (84.51%) and specialized methods like CleanNet.
- Clothing1M: In a purely noisy training setting, AFM reached 74.12% accuracy, a significant improvement over the standard Softmax baseline. With additional clean-set fine-tuning, it set a new benchmark at 82.09%.
- Efficiency: AFM is highly efficient, adding negligible training time (e.g., 24 seconds to a 4-hour run on Food101N).
Conclusion
AFM offers a lightweight and effective strategy for training DNNs on noisy data. By leveraging grouping and self-attention, it avoids the need for clean validation sets or complex noise assumptions. While the current implementation focuses on training-time suppression, future work could explore extending this attentive logic to inference or semi-supervised learning tasks.
Summary updated