On Robust Learning from Noisy Labels: A Permutation Layer Approach

Salman Al-Subaihi, Mohammed Alkhrashi, Raied Aljadaany, Fahad Albalawi, Bernard GhanemarXiv 2022 · DOI · arXiv

Abstract

Deep Neural Networks (DNNs) are highly susceptible to the "memorization effect," where they eventually fit noisy labels, degrading generalization. This paper introduces PermLL (Permutation Layer Learning), a framework that integrates a learnable, instance-dependent permutation layer into the training process. By permuting model predictions to match noisy labels, PermLL effectively cleans the loss signal. Experimental results on CIFAR and the real-world Clothing1M dataset demonstrate that PermLL achieves state-of-the-art performance, particularly in high-noise scenarios, by leveraging confidence-aware updates.

Introduction

Scaling deep learning requires massive datasets, often sourced from noisy channels like web scraping or non-expert crowdsourcing. While DNNs are powerful, they tend to memorize these incorrect labels. Existing solutions typically utilize a global "transition matrix" to model label flips, but these assume noise is instance-independent—ignoring the fact that a confusing image is more likely to be mislabeled than a clear one. PermLL addresses this by providing a simple, theoretically grounded, instance-dependent correction mechanism.

Methodology: The Permutation Layer

The core innovation is a learnable Permutation Layer. For each training sample, the model maintains a parameter $\alpha$ defining a permutation matrix $\bar{P}_\alpha$, which is a convex combination of elementary permutations. The authors investigate two primary architectural integrations:

  1. Permuting Labels: The layer is applied to the noisy label $\tilde{y}$ to "clean" it before calculating the loss.
  2. Permuting Predictions: The layer is applied to the model’s output $f_\theta(x)$ to "noisify" the prediction so it aligns with the noisy label during training.
Figure 1: Comparison between Permuting Labels (left) and Permuting Predictions (right) architectures.

The Advantage of Permuting Predictions

Theoretical analysis reveals that permuting predictions is significantly more robust. While permuting labels often leads to premature convergence at local minima, permuting predictions is "confidence-aware." The gradients for the permutation layer are naturally scaled by the model's certainty: if the model is unsure, it makes minimal changes; if it is highly confident but the label is contradictory, it aggressively learns to flip that label. This prevents over-correction during early training stages.

Experimental Results

PermLL was evaluated on synthetic benchmarks (CIFAR-10/100) and the real-world Clothing1M dataset (~38.5% noise).

  • High-Noise Robustness: On CIFAR-100 with 80% noise, PermLL achieved 45.81% accuracy, vastly outperforming the Cross-Entropy baseline (18.10%) and previous state-of-the-art methods like ELR (29.78%).
  • Real-World Performance: On Clothing1M, the method reached 74.99% accuracy, surpassing more complex frameworks like DivideMix despite its simpler implementation.
  • Training Dynamics: The framework successfully learns the correct permutations over time, as shown in the training curves where the percentage of correct permutations stabilizes at high levels depending on the learning rate $\eta_\alpha$ and initialization $I_\alpha$.
Figure 2: Training dynamics showing the percentage of correct permutations learned over epochs and sensitivity to hyperparameters.

Discussion and Limitations

PermLL simplifies robust learning by removing the need for complex "warm-up" periods or multi-network ensembles. However, some limitations remain. Its performance on specific asymmetric noise patterns (e.g., 40% on CIFAR-10) is slightly lower than specialized baselines. Additionally, the method is sensitive to the permutation learning rate $\eta_\alpha$ and initialization $I_\alpha$, requiring some tuning for optimal results. Finally, because the permutation layer is discarded at inference, the specific knowledge of noisy instances is only utilized to refine the classifier's weights during the training phase.

Summary updated

Abstract

The existence of label noise imposes significant challenges (e.g., poor generalization) on the training process of deep neural networks (DNN). As a remedy, this paper introduces a permutation layer learning approach termed PermLL to dynamically calibrate the training process of the DNN subject to instance-dependent and instance-independent label noise. The proposed method augments the architecture of a conventional DNN by an instance-dependent permutation layer. This layer is essentially a convex combination of permutation matrices that is dynamically calibrated for each sample. The primary objective of the permutation layer is to correct the loss of noisy samples mitigating the effect of label noise. We provide two variants of PermLL in this paper: one applies the permutation layer to the model's prediction, while the other applies it directly to the given noisy label. In addition, we provide a theoretical comparison between the two variants and show that previous methods can be seen as one of the variants. Finally, we validate PermLL experimentally and show that it achieves state-of-the-art performance on both real and synthetic datasets.
BibTeX
@misc{alsubaihi2022robust,
  author = {Salman Al-Subaihi and Mohammed Alkhrashi and Raied Aljadaany and Fahad Albalawi and Bernard Ghanem},
  title = {{On Robust Learning from Noisy Labels: A Permutation Layer Approach}},
  year = {2022},
  eprint = {2211.15890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15890},
}
Figure