Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

Junjie Liu, Dongchao Wen, Hongxing Gao, Wei Tao, Tse-Wei Chen, Kinya Osa, Masami KatoWorkshops @ CVPR 2019 · DOI

Introduction

Knowledge Distillation (KD) is a cornerstone technique for model compression, enabling small "student" networks to inherit the performance of large, pre-trained "teacher" models. However, traditional KD methods often struggle with two primary issues: the capacity gap, where the student lacks the representational power to mimic complex teacher boundaries, and residual accumulation, where optimization errors compound across deep teacher layers.

The "Knowledge Representing" (KR) framework addresses these challenges by shifting the focus from mimicking outputs (posterior knowledge) to efficiently representing internal parameters (prior knowledge). By aggregating teacher layers and applying a novel gradient-based sparsity penalty, KR enables tiny models to achieve state-of-the-art performance with significantly faster convergence.

The Knowledge Representing Framework

The KR framework introduces two innovative components designed to make teacher knowledge more "digestible" for smaller architectures.

Overview of the Knowledge Representing (KR) framework, illustrating knowledge aggregation from teacher layers and sparse recoding for the student network.

Knowledge Aggregation Scheme

To mitigate residual accumulation, the authors avoid layer-for-layer matching. Instead, they aggregate multiple neighboring teacher layers into a single, abstract representation. This process uses Optimal Transportation theory to minimize the cost of mapping high-dimensional teacher knowledge into the student's more constrained parameter space. This abstraction allows the student to learn a robust "summary" of the teacher's logic rather than struggling to replicate every intermediate step.

Sparse Recoding Penalty (SP)

Unlike standard $L_1$ or $L_2$ regularization that acts on weights, the Sparse Recoding Penalty (SP) targets the gradients during training. Using a learnable threshold ($\epsilon$), the penalty smooths low-magnitude gradients to zero—filtering out noise—while amplifying dominant gradients. This forces the student to focus on the most critical features, resulting in a highly discriminative parameter distribution.

Experimental Results

The KR framework was rigorously tested across standard benchmarks, including CIFAR-10, CIFAR-100, and ImageNet (ILSVRC 2012).

Performance and Convergence

On CIFAR-10, a ResNet-8 student using KR achieved 90.65% accuracy (up from an 87.91% baseline). When combined with KDGAN, it reached 91.35%, nearly matching the ResNet-26 teacher (91.91%) despite having only a fraction of the parameters. On ImageNet, the method outperformed established techniques like FitNet and NTS with a Top-1 error of 23.47%.

Training loss comparison across different regularization methods, showing that the SP penalty leads to faster convergence and lower final loss.

A key finding is the impact on training efficiency. As shown in the loss curves, the SP penalty facilitates a "bigger step" for gradient descent early in training, leading to significantly faster convergence compared to $L_1$ or $L_2$ norms.

Generalization and Robustness

To test generalization, the authors introduced TCIFAR-100, where images are distorted with Gaussian artifacts. The KR-trained student achieved 60.03% accuracy, significantly outperforming the 54.37% baseline. This suggests that the aggregated, sparse representations learned via KR are more resilient to input noise and distribution shifts.

Conclusion and Future Work

The Knowledge Representing framework demonstrates that how we represent prior knowledge is as important as the distillation process itself. By focusing on abstraction and gradient-level sparsity, KR bridges the capacity gap between large and small models. Future research will explore the framework's sensitivity to initialization and its performance in non-vision tasks like natural language processing, as well as its compatibility with low-bit quantization.

Summary updated

BibTeX
@inproceedings{liu2019knowledge,
  author = {Junjie Liu and Dongchao Wen and Hongxing Gao and Wei Tao and Tse-Wei Chen and Kinya Osa and Masami Kato},
  title = {{Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00090},
}
Figure