TL;DR
This paper introduces CRUST, a novel method for training deep neural networks that are robust to noisy labels. CRUST identifies and trains on carefully selected subsets of "clean" data points, providing theoretical guarantees against overfitting and achieving state-of-the-art performance on challenging noisy datasets.
What Problem Does This Paper Address?
Deep neural networks are incredibly powerful, but their performance is highly dependent on the quality of the training data. In real-world scenarios, obtaining perfectly labeled datasets is often prohibitively expensive, leading to the ubiquitous presence of noisy labels (incorrectly labeled examples). Deep networks have a notorious ability to "memorize" these noisy labels, severely degrading their generalization performance. While many empirical methods exist to combat this, they often lack theoretical guarantees. This paper aims to bridge this gap by proposing a method that not only achieves robust training but also provides a theoretical foundation for its effectiveness.
Methods
The core idea behind CRUST is to leverage recent theoretical insights into the training dynamics of neural networks. Specifically, the authors build upon the observation that the Jacobian matrix of a neural network during training has a structure where learning along directions associated with large singular values is fast and generalizes well, while learning along directions with small singular values leads to overfitting. Noisy labels tend to align with these "nuisance" directions.
CRUST addresses this by:
- Identifying Clean Data Subsets (Coresets): The method iteratively selects subsets of data points that are likely to be clean and contribute to an approximately low-rank Jacobian matrix. This is achieved by framing the problem as a k-medoids clustering task in the gradient space. The intuition is that clean data points with similar gradients will cluster together, while noisy points will be more scattered.
- Weighted Gradient Descent: The selected subsets are then used for training via weighted gradient descent. The weights are derived from the size of the clusters each selected data point represents, effectively giving more importance to the "central" points of clean clusters.
- Mixup for Error Reduction: To further mitigate errors arising from imperfect coreset selection and estimation, CRUST incorporates a mixup strategy. This involves interpolating between the selected "center" data points and a few other points from their respective clusters, effectively smoothing the training distribution and reducing the impact of any remaining noisy labels.
Datasets
The paper evaluates CRUST on two main types of datasets:
- Artificially Corrupted Datasets: CIFAR-10 and CIFAR-100 were used with both symmetric and asymmetric label noise introduced at various ratios (20%, 50%, 80%).
- Real-World Noisy Dataset: mini WebVision, a subset of a larger dataset containing naturally occurring noisy labels from web-crawled images.
Key Insights and Ideas
- Jacobian Structure is Key: The paper strongly emphasizes that understanding the singular value decomposition of the neural network's Jacobian matrix is crucial for understanding and combating label noise.
- Gradient Space Clustering as a Proxy for Cleanliness: The novel idea of using k-medoids clustering in the gradient space to identify likely clean data points is a significant contribution. This approach is more computationally tractable than directly analyzing the Jacobian's rank.
- Theoretical Guarantees for Robustness: Unlike many empirical methods, CRUST is backed by theoretical proofs that demonstrate its ability to prevent overfitting to noisy labels under certain conditions.
- Mixup as a Practical Enhancement: The integration of mixup, a technique known for improving generalization, is shown to be effective in further reducing the impact of residual noise in the selected coresets.
Key Results
- State-of-the-Art Performance: CRUST consistently outperforms existing state-of-the-art baselines across various noisy settings.
- Significant Accuracy Gains: On CIFAR-10 with 80% symmetric noisy labels, CRUST achieved a 6% increase in accuracy compared to the strongest baseline (INCV). On mini WebVision, it achieved a 7% increase in top-1 accuracy.
- Robustness Across Noise Types: The method demonstrates strong performance on both symmetric and asymmetric label noise.
- Efficiency: CRUST achieves its performance gains without requiring auxiliary models or extra clean datasets, and its training process is competitive with or faster than training on the entire dataset.
Limitations and Future Work
- Theoretical Proofs for Generalization: While the paper provides theoretical guarantees against overfitting, a formal proof for generalization performance is deferred to future work.
- Computational Cost of Jacobian: Although CRUST aims to reduce computational burden, calculating pairwise gradient dissimilarities still requires backpropagation on the entire dataset, which can be expensive for very large models and datasets. The paper mentions using upper bounds to mitigate this.
- Coreset Size Optimization: The paper investigates the effect of coreset size, noting that too small a coreset might not capture the information space well, while too large a coreset might include too much noise. Finding the optimal size remains an empirical consideration.
- Further Exploration of Mixup Hyperparameters: The paper uses a fixed mixup hyperparameter and suggests that further tuning could potentially yield even better results.
Summary updated