In standard classification tasks, we typically train models using "one-hot" labels (e.g., a 1 for "dog" and 0 for everything else). However, real-world data is messy. A photo might be ambiguous, or the label might simply be wrong. While techniques like Knowledge Distillation (KD)—where a "student" model learns from a "teacher" model's soft probabilities—often perform better than training on raw labels, the exact reasons why this works have remained somewhat mysterious.
This paper seeks to quantify what makes a "good" supervisory signal and explains how a model spontaneously discovers better labels during its own training process. By observing these learning paths, the authors introduce Filter-KD, a distillation method that extracts a cleaner, more stable supervisory signal, significantly boosting model accuracy and calibration.
The Criterion for Good Supervision
The authors propose a simple criterion: a supervisory signal is "good" if it is close in $L_2$ distance to the true, ideal categorical distribution ($p^*$). To study this, they track the learning path—the trajectory of a model’s predictions for a specific sample throughout the training process.
By tracking these paths, the researchers observed a phenomenon they call the "zig-zag" learning path. For "hard" or mislabeled samples, the model first moves toward the ground truth ($p^*$) because it is influenced by "similar" samples in the dataset. Only later in training does it "veer off" to memorize the specific (and potentially incorrect) one-hot label.
The paper explains this zig-zag path via the Neural Tangent Kernel (NTK). Early in training, a sample's prediction is driven by its "neighbors." If most neighbors are correct, they pull the prediction toward the true distribution. Later, the specific loss for that individual sample takes over, forcing the model to memorize the one-hot label even if it's "bad." In essence, models are naturally "smarter" in the middle of training than at the end; they "know" a label is likely wrong before they are forced to memorize it.
Filter-KD: Smoothing the Path
Because teacher signals near the end of training are often unstable, relying on a single snapshot of a teacher (like standard KD or Early-Stopped KD) is suboptimal.
Inspired by the zig-zag observation, the authors propose Filter-KD. Instead of using a single snapshot, they maintain a running average (Exponential Moving Average) of the teacher's predictions for every sample. This acts as a low-pass filter, smoothing out the high-frequency "noise" of the learning path and capturing the model's "consensus" on what the label should be. This provides a much more stable and reliable "dark knowledge" for the student to learn from.
Experimental Validation and Key Results
The authors validated their findings across several benchmarks, including a Toy Gaussian Dataset, CIFAR-10/100, CIFAR-10H, and TinyImageNet, as well as noisy variants where 5% to 20% of labels were randomly flipped.
Key findings include:
- Robustness to Noise: On TinyImageNet with 20% noisy labels, Filter-KD achieved 40.81% accuracy, compared to just 30.52% for standard one-hot training and 36.55% for Early-Stopped KD.
- Superior Calibration: Filter-KD doesn't just improve accuracy; it dramatically improves calibration (how well the model's confidence matches its actual accuracy). On clean CIFAR-10, Filter-KD reduced the Expected Calibration Error (ECE) to 0.007, a massive improvement over standard training (0.026) and standard KD (0.027).
- Self-Distillation Gains: Even without a larger teacher, Filter-KD consistently outperformed standard self-distillation across all tested architectures (ResNet, EfficientNet, MobileNetV2).
Limitations and Future Work
While highly effective, Filter-KD does introduce some overhead:
- Memory Overhead: The method requires storing a "lookup table" of moving averages for the entire dataset ($N$ samples $\times$ $K$ classes). For massive datasets with millions of images and thousands of classes, this could become memory-intensive.
- Parameter Space vs. Function Space: The authors tried filtering the model's weights (similar to Stochastic Weight Averaging) instead of its outputs, but found it wasn't as effective for refining labels. They suggest further research into why "function space" filtering is superior here.
- Hyperparameter Sensitivity: While a temperature ($\tau$) of 1 works well, the smoothing factor ($\alpha$) still needs to be tuned. The paper provides a grid search suggesting 0.05 is a robust default, but optimal values may vary by task.
Summary updated