Introduction
Knowledge Distillation (KD) is a cornerstone technique for model compression, enabling small "student" networks to inherit the performance of large, pre-trained "teacher" models. However, traditional KD methods often struggle with two primary issues: the capacity gap, where the student lacks the representational power to mimic complex teacher boundaries, and residual accumulation, where optimization errors compound across deep teacher layers.
The "Knowledge Representing" (KR) framework addresses these challenges by shifting the focus from mimicking outputs (posterior knowledge) to efficiently representing internal parameters (prior knowledge). By aggregating teacher layers and applying a novel gradient-based sparsity penalty, KR enables tiny models to achieve state-of-the-art performance with significantly faster convergence.
The Knowledge Representing Framework
The KR framework introduces two innovative components designed to make teacher knowledge more "digestible" for smaller architectures.
Knowledge Aggregation Scheme
To mitigate residual accumulation, the authors avoid layer-for-layer matching. Instead, they aggregate multiple neighboring teacher layers into a single, abstract representation. This process uses Optimal Transportation theory to minimize the cost of mapping high-dimensional teacher knowledge into the student's more constrained parameter space. This abstraction allows the student to learn a robust "summary" of the teacher's logic rather than struggling to replicate every intermediate step.
Sparse Recoding Penalty (SP)
Unlike standard $L_1$ or $L_2$ regularization that acts on weights, the Sparse Recoding Penalty (SP) targets the gradients during training. Using a learnable threshold ($\epsilon$), the penalty smooths low-magnitude gradients to zero—filtering out noise—while amplifying dominant gradients. This forces the student to focus on the most critical features, resulting in a highly discriminative parameter distribution.
Experimental Results
The KR framework was rigorously tested across standard benchmarks, including CIFAR-10, CIFAR-100, and ImageNet (ILSVRC 2012).
Performance and Convergence
On CIFAR-10, a ResNet-8 student using KR achieved 90.65% accuracy (up from an 87.91% baseline). When combined with KDGAN, it reached 91.35%, nearly matching the ResNet-26 teacher (91.91%) despite having only a fraction of the parameters. On ImageNet, the method outperformed established techniques like FitNet and NTS with a Top-1 error of 23.47%.
A key finding is the impact on training efficiency. As shown in the loss curves, the SP penalty facilitates a "bigger step" for gradient descent early in training, leading to significantly faster convergence compared to $L_1$ or $L_2$ norms.
Generalization and Robustness
To test generalization, the authors introduced TCIFAR-100, where images are distorted with Gaussian artifacts. The KR-trained student achieved 60.03% accuracy, significantly outperforming the 54.37% baseline. This suggests that the aggregated, sparse representations learned via KR are more resilient to input noise and distribution shifts.
Conclusion and Future Work
The Knowledge Representing framework demonstrates that how we represent prior knowledge is as important as the distillation process itself. By focusing on abstraction and gradient-level sparsity, KR bridges the capacity gap between large and small models. Future research will explore the framework's sensitivity to initialization and its performance in non-vision tasks like natural language processing, as well as its compatibility with low-bit quantization.
Summary updated