Suppressing Mislabeled Data via Grouping and Self-attention

Xiaojiang Peng, Kai Wang, Zhaoyang Zeng, Qing Li, Jianfei Yang, Yu Qiao

Abstract

Deep Neural Networks (DNNs) are highly susceptible to memorizing noisy labels in large-scale, web-scraped datasets, which severely compromises their generalization capabilities. This paper proposes Attentive Feature Mixup (AFM), a novel training module designed to identify and down-weight mislabeled samples dynamically. By grouping samples and applying a self-attention mechanism, AFM generates "noise-suppressed" synthetic samples that provide a cleaner training signal. AFM is an assumption-free, plug-and-play solution that achieves state-of-the-art results on real-world benchmarks like Food101N and Clothing1M without requiring auxiliary clean data.

Introduction

The success of DNNs is often predicated on the availability of massive, accurately labeled datasets. However, real-world data collection—such as scraping images based on tags—inevitably introduces label noise. Standard training procedures lead models to memorize these errors, destroying their ability to generalize to clean test data. While previous research has attempted to mitigate this through clean subsets or noise-modeling, these approaches are often limited by rigid assumptions or the need for expensive manual verification.

AFM addresses these challenges by treating noise detection as a relative problem: by comparing samples within small groups, the model can more easily identify "odd one out" instances that do not align with the group's dominant features.

Methodology: Attentive Feature Mixup (AFM)

The AFM block is integrated into the training pipeline and consists of two primary components: the Group-to-Attend (GA) module and the Mixup module.

The AFM architecture, illustrating the Group-to-Attend (GA) module where samples interact to produce attention weights, followed by the Mixup module for synthetic sample generation.

Group-to-Attend (GA) Module

In each mini-batch, the GA module randomly selects a small group of $K$ samples (typically $K=2$). These samples are projected into an embedding space via separate fully-connected layers ($FC_a$ and $FC_b$). The embeddings interact—most effectively through summation—to allow the network to compare features. A lightweight self-attention network then assigns a weight ($\alpha$) to each sample. The objective is for the network to learn to assign higher weights to clean, representative samples and lower weights to those suspected of being mislabeled.

Mixup and Multi-task Learning

The Mixup module uses the computed attention weights to perform a weighted interpolation of the group's features and labels, creating a "noise-suppressed" virtual sample. The model is trained using a multi-task loss: one classifier ($f_{c2}$) processes the original noisy samples, while another ($f_{c1}$) processes the synthetic, cleaner samples. This dual-path approach ensures the model learns robust features from the mixed data while still seeing the full dataset.

Theoretical Motivation and Noise Dilution

The effectiveness of AFM stems from the principle of "noise dilution." While a dataset may have a high overall noise ratio, the probability of selecting a group where every member is mislabeled is significantly lower. By mixing samples, the influence of a single noisy label is diluted by the presence of clean data in the same group.

Conceptual diagram showing how AFM facilitates boundary expansion and handles various noise types (inter-class and intra-class) through attentive mixing.

As shown in Figure 1, this process effectively expands the decision boundaries of the classes. The authors also demonstrate that for the attention weights to be discriminative, the projection layers for group members must be distinct ($f_a \neq f_b$), ensuring the network can correctly associate weights with specific samples.

Experimental Results

The authors evaluated AFM on two large-scale, real-world noisy datasets: Food101N (365k images, ~20% noise) and Clothing1M (1M images, ~38.5% noise).

  • Food101N: AFM achieved a top-1 accuracy of 87.23%, outperforming the ResNet50 baseline (84.51%) and specialized methods like CleanNet.
  • Clothing1M: In a purely noisy training setting, AFM reached 74.12% accuracy, a significant improvement over the standard Softmax baseline. With additional clean-set fine-tuning, it set a new benchmark at 82.09%.
  • Efficiency: AFM is highly efficient, adding negligible training time (e.g., 24 seconds to a 4-hour run on Food101N).

Conclusion

AFM offers a lightweight and effective strategy for training DNNs on noisy data. By leveraging grouping and self-attention, it avoids the need for clean validation sets or complex noise assumptions. While the current implementation focuses on training-time suppression, future work could explore extending this attentive logic to inference or semi-supervised learning tasks.

Summary updated

Abstract

Deep networks achieve excellent results on large-scale clean data but degrade significantly when learning from noisy labels. To suppressing the impact of mislabeled data, this paper proposes a conceptually simple yet efficient training block, termed as Attentive Feature Mixup (AFM), which allows paying more attention to clean samples and less to mislabeled ones via sample interactions in small groups. Specifically, this plug-and-play AFM first leverages a group-to-attend module to construct groups and assign attention weights for group-wise samples, and then uses a mixup module with the attention weights to interpolate massive noisy-suppressed samples. The AFM has several appealing benefits for noise-robust deep learning. (i) It does not rely on any assumptions and extra clean subset. (ii) With massive interpolations, the ratio of useless samples is reduced dramatically compared to the original noisy ratio. (iii) It jointly optimizes the interpolation weights with classifiers, suppressing the influence of mislabeled data via low attention weights. (iv) It partially inherits the vicinal risk minimization of mixup to alleviate over-fitting while improves it by sampling fewer feature-target vectors around mislabeled data from the mixup vicinal distribution. Extensive experiments demonstrate that AFM yields state-of-the-art results on two challenging real-world noisy datasets: Food101N and Clothing1M.
BibTeX
@inproceedings{peng2020suppressing,
  author = {Xiaojiang Peng and Kai Wang and Zhaoyang Zeng and Qing Li and Jianfei Yang and Yu Qiao},
  title = {{Suppressing Mislabeled Data via Grouping and Self-attention}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58517-4\_46},
}
Figure