Abstract
Deep Neural Networks (DNNs) are highly susceptible to the "memorization effect," where they eventually fit noisy labels, degrading generalization. This paper introduces PermLL (Permutation Layer Learning), a framework that integrates a learnable, instance-dependent permutation layer into the training process. By permuting model predictions to match noisy labels, PermLL effectively cleans the loss signal. Experimental results on CIFAR and the real-world Clothing1M dataset demonstrate that PermLL achieves state-of-the-art performance, particularly in high-noise scenarios, by leveraging confidence-aware updates.
Introduction
Scaling deep learning requires massive datasets, often sourced from noisy channels like web scraping or non-expert crowdsourcing. While DNNs are powerful, they tend to memorize these incorrect labels. Existing solutions typically utilize a global "transition matrix" to model label flips, but these assume noise is instance-independent—ignoring the fact that a confusing image is more likely to be mislabeled than a clear one. PermLL addresses this by providing a simple, theoretically grounded, instance-dependent correction mechanism.
Methodology: The Permutation Layer
The core innovation is a learnable Permutation Layer. For each training sample, the model maintains a parameter $\alpha$ defining a permutation matrix $\bar{P}_\alpha$, which is a convex combination of elementary permutations. The authors investigate two primary architectural integrations:
- Permuting Labels: The layer is applied to the noisy label $\tilde{y}$ to "clean" it before calculating the loss.
- Permuting Predictions: The layer is applied to the model’s output $f_\theta(x)$ to "noisify" the prediction so it aligns with the noisy label during training.
The Advantage of Permuting Predictions
Theoretical analysis reveals that permuting predictions is significantly more robust. While permuting labels often leads to premature convergence at local minima, permuting predictions is "confidence-aware." The gradients for the permutation layer are naturally scaled by the model's certainty: if the model is unsure, it makes minimal changes; if it is highly confident but the label is contradictory, it aggressively learns to flip that label. This prevents over-correction during early training stages.
Experimental Results
PermLL was evaluated on synthetic benchmarks (CIFAR-10/100) and the real-world Clothing1M dataset (~38.5% noise).
- High-Noise Robustness: On CIFAR-100 with 80% noise, PermLL achieved 45.81% accuracy, vastly outperforming the Cross-Entropy baseline (18.10%) and previous state-of-the-art methods like ELR (29.78%).
- Real-World Performance: On Clothing1M, the method reached 74.99% accuracy, surpassing more complex frameworks like DivideMix despite its simpler implementation.
- Training Dynamics: The framework successfully learns the correct permutations over time, as shown in the training curves where the percentage of correct permutations stabilizes at high levels depending on the learning rate $\eta_\alpha$ and initialization $I_\alpha$.
Discussion and Limitations
PermLL simplifies robust learning by removing the need for complex "warm-up" periods or multi-network ensembles. However, some limitations remain. Its performance on specific asymmetric noise patterns (e.g., 40% on CIFAR-10) is slightly lower than specialized baselines. Additionally, the method is sensitive to the permutation learning rate $\eta_\alpha$ and initialization $I_\alpha$, requiring some tuning for optimal results. Finally, because the permutation layer is discarded at inference, the specific knowledge of noisy instances is only utilized to refine the classifier's weights during the training phase.
Summary updated