SuperLoss: A Generic Loss for Robust Curriculum Learning

Thibault Castells, Philippe Weinzaepfel, Jérôme RevaudNeurIPS 2020 · OpenReview

Overview

SuperLoss is a simple, "plug-and-play" loss function that can be appended to any existing task loss to automatically enable curriculum learning. By dynamically downweighting samples with high loss—which are likely "hard" or noisy—it improves model generalization and provides remarkable robustness against label noise across classification, regression, and object detection tasks.

The Challenge of Curriculum Learning

Curriculum Learning (CL) is based on the intuitive idea that models learn better when presented with "easy" examples before "hard" ones, much like human education. However, implementing CL in deep learning is traditionally difficult for three main reasons:

  1. Task Specificity: Most methods are tailored specifically for classification and do not easily translate to regression or object detection.
  2. Complexity: Existing approaches often require multi-stage training pipelines, extra learnable parameters, or specialized "warm-up" periods.
  3. Noise Sensitivity: Standard loss functions (like Mean Squared Error or Cross-Entropy) are highly sensitive to outliers and noisy labels. Models often try to "memorize" these bad examples, which degrades overall performance.

The authors sought a task-agnostic, simple method to achieve the benefits of curriculum learning without fundamentally altering the underlying training pipeline.

The SuperLoss Method

The core contribution of this work is SuperLoss (SL). Instead of modifying the neural network architecture or creating a complex multi-stage training regimen, the authors propose adding a "loss on top of a loss."

Training pipeline comparison showing SuperLoss appended to the standard loss
  1. Confidence-Aware Formulation: They define a new loss function $L_\lambda(\ell_i, \sigma_i)$ that takes the original task loss ($\ell_i$) and a confidence parameter ($\sigma_i$) as inputs.
  2. The Formula: $L_\lambda(\ell_i, \sigma_i) = (\ell_i - \tau)\sigma_i + \lambda(\log \sigma_i)^2$.
  3. $\tau$ is a threshold separating "easy" from "hard" samples (often calculated as a running average of the loss).
  4. $\sigma_i$ represents the model's confidence in that specific sample.
  5. $\lambda$ is a regularization hyperparameter.
  6. Optimal Confidence: Rather than learning $\sigma_i$ through slow backpropagation—which adds memory overhead—the authors derive a closed-form solution for the optimal confidence using the Lambert W function.
  7. Dynamic Downweighting: During training, if a sample has a very high loss (much larger than $\tau$), its optimal confidence $\sigma_i$ drops toward zero. This effectively flattens the gradient for that sample, preventing the model from over-adjusting to potential outliers or extremely difficult cases.

Key Insights and Advantages

  • The "Agnostic" Advantage: Because SuperLoss only looks at the scalar value of the input loss, it is completely agnostic to the task. It works whether that loss came from a bounding box regressor, a softmax classifier, or a triplet loss for embeddings.
  • Implicit Curriculum: By downweighting high-loss samples, the model naturally focuses on "easy" samples early in training. As the model improves and the average loss ($\tau$) drops, it gradually begins to incorporate more difficult samples.
  • Noise as "Infinite Hardness": Noisy samples (e.g., a picture of a cat labeled "toaster") will consistently produce a high loss. SuperLoss treats these as "permanently hard" and keeps their weights low, preventing the model from memorizing the noise.
  • No Extra Parameters: Unlike previous "Self-Paced Learning" methods that required storing a weight for every single image in a dataset, SuperLoss calculates the weight on-the-fly, making it scalable to massive datasets.

Experimental Results

The authors evaluated SuperLoss across a wide variety of benchmarks to prove its generality:

  • Regression: On the UTKFace age regression dataset with 80% artificial noise, SuperLoss reduced the Mean Absolute Error (MAE) from 17.56 down to 13.09.
  • Object Detection: In Pascal VOC with 60% label noise, SuperLoss improved the mAP (Mean Average Precision) by 5 points for Faster R-CNN and 9 points for RetinaNet compared to standard training.
  • Classification: On CIFAR-10 and CIFAR-100, SuperLoss performed on par with or better than complex, specialized noise-robustness frameworks, despite being significantly simpler to implement.
  • Large-Scale Real-World Noise: On the massive WebVision dataset (which contains natural, semi-automatically collected noise), simply adding SuperLoss provided a +1.2% top-1 accuracy boost with zero changes to training time or engineering effort.

Limitations and Future Work

While highly effective, SuperLoss does introduce two hyperparameters: $\lambda$ (regularization) and the choice of how to calculate the threshold $\tau$. While the authors suggest using a running average for $\tau$, some tuning is still required for optimal results.

Additionally, there is an inherent "easy bias." In some scenarios, "hard" samples are actually the most informative. If $\lambda$ is set too aggressively, the model might ignore legitimately difficult but correct samples, potentially limiting its ultimate performance ceiling on very complex, clean datasets. Future work may explore more theoretical grounding for the "optimal" thresholding strategy to balance this tradeoff.

Summary updated

Abstract

Curriculum learning is a technique to improve a model performance and generalization based on the idea that easy samples should be presented before difficult ones during training. While it is generally complex to estimate a priori the difficulty of a given sample, recent works have shown that curriculum learning can be formulated dynamically in a self-supervised manner. The key idea is to somehow estimate the importance (or weight) of each sample directly during training based on the observation that easy and hard samples behave differently and can therefore be separated. However, these approaches are usually limited to a specific task (e.g., classification) and require extra data annotations, layers or parameters as well as a dedicated training procedure. We propose instead a simple and generic method that can be applied to a variety of losses and tasks without any change in the learning procedure. It consists in appending a novel loss function on top of any existing task loss, hence its name: the SuperLoss. Its main effect is to automatically downweight the contribution of samples with a large loss, i.e. hard samples, effectively mimicking the core principle of curriculum learning. As a side effect, we show that our loss prevents the memorization of noisy samples, making it possible to train from noisy data even with non-robust loss functions. Experimental results on image classification, regression, object detection and image retrieval demonstrate consistent gain, particularly in the presence of noise.
BibTeX
@inproceedings{castells2020superloss,
  author = {Thibault Castells and Philippe Weinzaepfel and Jérôme Revaud},
  title = {{SuperLoss: A Generic Loss for Robust Curriculum Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=huSo2tHAhM},
}
Figure