Abstract
Deep neural networks are often too computationally expensive for edge devices, leading to the use of Knowledge Distillation (KD) to create smaller "student" models. However, these compact models frequently lack robustness against real-world noise. This paper proposes a novel framework for Robust Knowledge Distillation that optimizes for higher prediction confidence and gradient alignment with a teacher network. The resulting models maintain high compression ratios while significantly outperforming standard distillation methods under noisy conditions.
Introduction
While large-scale neural networks achieve state-of-the-art performance, their deployment on mobile and IoT devices is limited by hardware constraints. Knowledge Distillation (KD) has emerged as a standard technique to transfer knowledge from a heavy "teacher" network to a lightweight "student" network. Despite its success in maintaining accuracy on clean datasets, existing KD frameworks often overlook model robustness. In real-world scenarios, data is frequently corrupted by sensor noise, motion blur, or occlusions, which can cause the performance of tiny student networks to collapse. This research introduces a method to bridge this "robustness gap."
Methodology
The authors propose a training objective derived from a theoretical analysis of the lower bound of perturbation—the minimum noise required to alter a network's prediction. To enhance this lower bound, two primary constraints are introduced:
- Confidence Constraint ($L_S$): This loss encourages the student network to produce a prediction score for the ground-truth label that exceeds the teacher's score by a margin $\gamma$. By making the student more "certain" than its teacher, the model becomes more resilient to fluctuations caused by noise.
- Gradient Constraint ($L_G$): This forces the student’s gradients with respect to the input image to align with the teacher’s gradients. Since gradients characterize a model's sensitivity to small pixel changes, this alignment ensures the student inherits the teacher's stability against input disturbances.
The final training objective is a weighted combination of standard KD loss, the confidence constraint, and the gradient constraint.
Experimental Setup
The proposed method was evaluated across several standard benchmarks:
- MNIST: Handwritten digits for baseline robustness testing.
- CIFAR-10 & CIFAR-100: Natural images used to test performance across varying category counts.
- USPS: Used for domain adaptation tests, where models trained on MNIST were evaluated on USPS to measure resilience to data distribution shifts.
Key Results
The experimental results demonstrate that the Robust Student network consistently maintains higher accuracy as noise levels increase compared to standard KD and FitNet approaches.
- Noise Resilience: On MNIST with high Gaussian noise (SNR=1), the Robust Student maintained 98.17% accuracy, while standard KD and FitNet dropped significantly lower.
- Efficiency and Compression: The architecture achieved a 36x compression ratio and a 13.17x speedup relative to the teacher, while staying within 1% of the teacher's accuracy on clean CIFAR-10 data.
- Occlusion and Domain Adaptation: The model showed superior performance when handling partial image occlusions and achieved 95.02% accuracy in the MNIST-to-USPS domain adaptation task, outperforming standard KD (93.25%).
Discussion and Limitations
The core insight of this work is that robustness requires mimicking not just the "what" (output labels) but the "how" (sensitivity to input changes). By creating a numerical confidence buffer and aligning input gradients, the student network becomes a more faithful and stable representation of the teacher.
However, there are trade-offs. Calculating input gradients during training ($L_G$) increases computational overhead, making training slower even though inference remains fast. Additionally, while the method excels against random noise and occlusions, its effectiveness against targeted adversarial attacks remains a subject for future research. Future work may also explore using Neural Architecture Search (NAS) to automate the design of these robust student structures.
Summary updated