Abstract
Deep neural networks are highly susceptible to memorizing noisy labels in large-scale datasets, which degrades their generalization performance. We introduce Attentive Feature MixUp (AFM), a novel, plug-and-play training block that enables models to suppress mislabeled data by learning from sample comparisons. AFM groups samples and utilizes a self-attention mechanism to assign higher weights to clean images and lower weights to noisy ones during feature interpolation. This process effectively "hallucinates" a cleaner training distribution. Our method achieves state-of-the-art performance on real-world noisy benchmarks, including Food101N and Clothing1M, while adding negligible computational cost.
Introduction
The success of deep learning is largely driven by massive labeled datasets. However, obtaining high-quality labels at scale is expensive and often impractical. Datasets scraped from the web frequently contain significant label noise. Standard training procedures lead networks to memorize these errors, harming their ability to generalize to unseen, clean data.
Existing approaches to handle label noise generally fall into three categories: noise-cleaning (removing or correcting labels), semi-supervised learning (using a small clean subset), and noise-robust methods (re-weighting individual samples). However, these methods often struggle to distinguish between "hard" clean samples and truly noisy ones, or they ignore the contextual relationships between different samples in a batch. AFM addresses these limitations by introducing a comparative logic, allowing the model to evaluate the reliability of a sample relative to others in a group.
The AFM Framework
The AFM block is designed to be integrated into existing CNN architectures. It consists of two primary components: the Group-to-Attend (GA) module and the MixUp module.
Group-to-Attend (GA) Module
Instead of processing images in isolation, AFM randomly groups $K$ samples (typically $K=2$) from a mini-batch. Each sample in the group is passed through a projection layer (a fully connected layer). A key theoretical finding of the paper is that these projection layers must be distinct for each slot in the group to ensure the attention mechanism remains effective.
The module then employs a self-attention network to compare the features within the group. By analyzing the interactions between features, the network learns to assign an attention weight ($\alpha$) to each sample. Ideally, clean samples are assigned high weights, while noisy samples receive low weights.
MixUp Module
Following the attention weight assignment, the module interpolates the features and labels of the group. The resulting "attentive-mixed" sample is a weighted combination of the group's inputs. If a group contains both a clean and a noisy sample, the attention mechanism ensures the clean sample dominates the resulting feature and label, effectively filtering out the noise. The final training loss combines the loss from original samples and these synthesized attentive-mixed samples.
Theoretical Foundations and Insights
The effectiveness of AFM is rooted in several key insights regarding data distribution and feature interaction.
Pure Noisy Reduction
Mathematically, the probability of a group of $K$ samples being entirely noisy is significantly lower than the probability of a single sample being noisy. By mixing samples, AFM reduces the frequency with which the model is forced to learn from purely incorrect information.
The Importance of Asymmetry
One of the more technical insights is Proposition 1: if you use a "sum" interaction to compare features, you must use different projection layers ($f_a \neq f_b$). Otherwise, the operation is commutative, the network cannot distinguish which weight belongs to which sample, and the attention mechanism becomes useless.
Vicinal Risk Minimization (VRM)
AFM can be viewed as an intelligent extension of standard MixUp. While standard MixUp blends samples randomly, AFM uses attention to prioritize samples from the "neighborhood" of clean data. This targeted interpolation helps in expanding decision boundaries more effectively than random mixing.
Experimental Evaluation
The authors evaluated AFM on two major real-world noisy benchmarks: Food101N (365k images, ~80% accuracy) and Clothing1M (1M images, ~61.5% accuracy).
- Performance: AFM set new records on both datasets. On Food101N, it achieved 87.23% accuracy, a 2% improvement over the previous state-of-the-art. On Clothing1M, it reached 82.09%.
- Efficiency: The computational overhead is minimal. For Food101N, AFM added only 24 seconds to a 4-hour training cycle on a ResNet50 baseline.
- Data Efficiency: AFM demonstrated remarkable robustness to reduced dataset sizes. Training on only 60% of Food101N with AFM yielded better results than other methods using the full dataset.
Limitations and Conclusion
While effective, AFM's performance is sensitive to the group size $K$. Increasing $K$ beyond 2 or 3 can lead to "over-smoothed" features, where the interpolated samples lose the discriminative characteristics necessary for learning distinct classes. Additionally, the choice of feature interaction remains an empirical decision.
In conclusion, Attentive Feature MixUp (AFM) provides a simple yet powerful way to improve the noise robustness of deep learning models. By leveraging grouping and self-attention, it allows models to distinguish between clean and noisy data through comparison. Future work may explore the application of AFM in semi-supervised settings to manage the noise inherent in pseudo-labeling.
Summary updated