Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum

Shreyas Saxena, Oncel Tuzel, Dennis DeCosteNeurIPS 2019 · OpenReview

In machine learning, the order in which data is presented matters. Just as human education builds from foundational concepts to complex theories, deep neural networks (DNNs) often generalize better when they tackle "easy" examples before "hard" ones. This strategy is known as Curriculum Learning.

However, defining what makes an image "easy" for a neural network is surprisingly difficult. Traditionally, researchers have relied on hand-crafted heuristics or complex multi-stage pipelines to define this difficulty. In a recent NeurIPS paper, researchers introduce a more elegant, automated solution: Data Parameters, a method that allows the model to learn its own curriculum using standard gradient descent.

The Challenge of Fixed Curricula

Curriculum learning is a powerful concept, but its traditional application is plagued by three main issues:

  1. Subjectivity: Defining "difficulty" for complex tasks is non-trivial and often relies on arbitrary human judgment.
  2. Human-AI Mismatch: What is easy for a human to identify might not be the easiest feature for a specific neural network architecture to learn first.
  3. Static Nature: A fixed curriculum doesn't account for the fact that a "hard" sample might become "easy" once the model has mastered certain sub-features.

The Solution: Learnable Data Parameters

To address these limitations, the authors introduce the Data Parameter ($\sigma$), a new set of learnable weights assigned to every class and individual sample in a dataset.

Instead of passing raw logits ($z$) directly into a standard Softmax function, the model scales the logits by the data parameter:

$$p_i = \frac{\exp(z_i / \sigma)}{\sum \exp(z_j / \sigma)}$$

This mechanism acts as learnable temperature scaling:

  • Small $\sigma$ (Easy Data): The probability distribution becomes sharper, amplifying the gradients for that sample. The model focuses its learning capacity on this data.
  • Large $\sigma$ (Hard/Noisy Data): The distribution flattens, decaying the gradients. The model effectively de-prioritizes or ignores this data.

During training, the model updates its standard weights to minimize loss while simultaneously updating the data parameters ($\sigma$) using Stochastic Gradient Descent (SGD). To maintain stability, optimization occurs in log-space with $L_2$ regularization, encouraging the model to eventually treat all data equally ($\sigma \approx 1$) unless specific samples are consistently problematic.

Key Insights and Dynamic Difficulty

The introduction of data parameters yields several fascinating behaviors during training:

  • Dynamic Difficulty: The curriculum is not a fixed schedule; it evolves. An image might be considered "hard" (high $\sigma$) early in training but become "easy" (low $\sigma$) as the model's feature extractors mature.
  • Automated Noise Filtering: When trained on datasets with heavy label noise, the model naturally learns to assign very high $\sigma$ values to corrupted samples. It effectively learns to ignore garbage data without any human intervention.
  • Zero Inference Overhead: Because data parameters are only used to scale logits during the training loss calculation, they are discarded once training is complete. The final model retains the exact same architecture and inference speed as a standard baseline model.
Evolution of class and instance parameters during training. The model dynamically adjusts difficulty, initially de-prioritizing unusual instances (orange line) before learning them later.

Performance and Results

The authors validated their approach across a wide variety of standard and challenging benchmarks, demonstrating consistent gains without increasing model complexity:

  • State-of-the-Art on Noisy Labels: On CIFAR-100 with 40% synthetic noise, the Data Parameter method achieved 70.93% accuracy, outperforming previous state-of-the-art methods like MentorNet (56.9%) and Robust Weighting (61.34%) by massive margins. It also showed strong performance on WebVision, a dataset crawled from the web with inherent label noise.
  • Scaling to Large Datasets: This work is among the first curriculum learning methods to show consistent gains on large-scale tasks, including ImageNet (+0.7% top-1 accuracy) and KITTI object detection (up to +1.3 mAP on "hard" instances).
  • Resistance to Memorization: When presented with a dataset of entirely random labels, standard DNNs eventually memorize the noise. The Data Parameter approach resisted this memorization, keeping training accuracy low by correctly identifying the random samples as "unlearnable."

Limitations and Future Directions

While highly effective, the method does introduce new considerations. Scaling logits during training can lead to uncalibrated probability outputs, requiring post-hoc calibration on a validation set. Additionally, the method introduces new hyperparameters (learning rate and weight decay for the $\sigma$ optimizer) that may require tuning for novel domains.

Looking forward, this framework holds significant promise beyond image classification and detection. The authors suggest it could be extended to tasks like Reinforcement Learning or Machine Translation, where determining the optimal "order of lessons" remains a notoriously difficult challenge.

Summary updated

Abstract

Recent works have shown that learning from easier instances first can help deep neural networks (DNNs) generalize better. However, knowing which data to present during different stages of training is a challenging problem. In this work, we address this problem by introducing data parameters. More specifically, we equip each sample and class in a dataset with a learnable parameter (data parameters), which governs their importance in the learning process. During training, at each iteration, as we update the model parameters, we also update the data parameters. These updates are done by gradient descent and do not require hand-crafted rules or design. When applied to image classification task on CIFAR10, CIFAR100,WebVision and ImageNet datasets, and object detection task on KITTI dataset, learning a dynamic curriculum via data parameters leads to consistent gains, without any increase in model complexity or training time. When applied to a noisy dataset, the proposed method learns to learn from clean images and improves over the state-of-the-art methods by 14%. To the best of our knowledge, our work is the first curriculum learning method to show gains on large scale image classification and detection tasks.
BibTeX
@inproceedings{saxena2019data,
  author = {Shreyas Saxena and Oncel Tuzel and Dennis DeCoste},
  title = {{Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=rPfhlI3aGEN},
}
Figure