Introduction
Data augmentation is a cornerstone of modern deep learning, essential for improving model generalization. Automated data augmentation (AutoAugment) has pushed this further by using machine learning to search for optimal transformation policies. However, aggressive transformations often create "noisy" images that lack clear semantic meaning, confusing the training process when paired with rigid ground-truth labels. This paper introduces a simple yet effective solution: using Knowledge Distillation (KD) to provide "soft labels" from a teacher model, allowing students to learn from ambiguous augmented data rather than being forced to fit incorrect one-hot labels.
The Problem: Augment Ambiguity
While AutoAugment significantly boosts performance, it suffers from a critical flaw known as Augment Ambiguity. When transformations become too aggressive—such as extreme shearing, brightness changes, or translations—the discriminative features of the object may disappear or shift significantly.
For instance, a "fox" might be shifted so far out of frame that the image is essentially empty, or an inverted color scheme might make it resemble a different species. Forcing a model to label such an image as a "fox" using a standard one-hot label introduces noise that destabilizes training and leads to under-fitting.
Proposed Method: Soft Supervision
The authors propose a hybrid supervision strategy that integrates a pre-trained "teacher" model into the training loop to evaluate every augmented image.
- Teacher-Student Framework: A teacher model provides a probability distribution (soft labels) for each augmented sample.
- Soft Labels: Instead of a hard "100% cat" label, the student matches the teacher’s nuanced output (e.g., "40% cat, 30% dog"), which better captures the ambiguity of distorted images.
- Top-K KL-Divergence: To filter irrelevant noise, the loss is only calculated based on the teacher's top-$K$ most confident classes.
- Combined Loss: The final training loss is a weighted sum of standard cross-entropy with ground-truth labels and the KL-divergence matching the teacher's soft labels.
Notably, the teacher does not need to be more powerful than the student; its value lies in acting as a semantic filter that judges whether an augmented sample still represents its original class.
Key Insights
The Under-fitting Trap
Without KD, increasing the magnitude of transformations eventually causes validation accuracy to crash. The model can no longer make sense of the "garbage" images it is forced to label, leading to severe under-fitting.
Rescuing Aggressive Augmentation
By using soft labels, researchers can utilize much higher transformation magnitudes. The soft labels "rescue" the model from noise, turning potentially harmful outliers into useful training signals. Furthermore, the study found that even a "weak" teacher can improve a "strong" student by providing better judgment of augmented samples than raw ground-truth labels can provide alone.
Results and Performance
The combination of AutoAugment and Knowledge Distillation yielded state-of-the-art (SOTA) results across major benchmarks:
- ImageNet: Achieved a top-1 accuracy of 85.8% using EfficientNet-B8 without extra training data.
- EfficientNet Improvements: Boosted EfficientNet-B7 from 84.9% to 85.5%.
- CIFAR-100: Reached a test error of 10.6%, outperforming competitors with similar training costs.
Conclusion
This research demonstrates that the limitations of automated data augmentation are not inherent to the transformations themselves, but rather to how we label the resulting data. By bridging data augmentation and distillation, the authors provide a robust framework for training high-performance models on increasingly complex and noisy datasets.
Summary updated