Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

Asit Mishra, Debbie MarrICLR 2018 · OpenReview

TL;DR

The "Apprentice" framework demonstrates that combining knowledge distillation with network quantization can significantly recover the accuracy lost when shrinking deep learning models. By using a high-precision "teacher" network to guide a low-precision "student," the authors achieve state-of-the-art results for ternary and 4-bit ResNet models on ImageNet, nearly matching full-precision performance.

The Challenge of Edge Deployment

Deep Neural Networks (DNNs) have revolutionized computer vision, but their success comes at a high cost: they are massive, compute-heavy, and memory-intensive. This makes deploying them on "edge" devices—like smartphones, drones, or IoT sensors—extremely difficult.

Engineers typically use two main strategies to slim down these models:

  1. Quantization: Reducing the precision of numbers (e.g., moving from 32-bit floating point to 2-bit ternary or 4-bit integers). This saves memory and speeds up hardware but usually tanks the model's accuracy.
  2. Knowledge Distillation (KD): Training a small "student" network to mimic the outputs (logits) of a large, pre-trained "teacher" network.

While both are popular, they were rarely used together effectively. The authors address the "accuracy gap" in low-precision networks, proving that KD is the "secret sauce" needed to make quantized models viable for production.

Methods: The Apprentice Framework

The authors introduce Apprentice, a framework that explores three distinct schemes for combining KD with quantization.

Knowledge Distillation Architecture
  • Scheme-A (Joint Training): Both the full-precision teacher and the low-precision student are trained from scratch simultaneously. Interestingly, the authors found that the teacher actually benefits from this, achieving slightly higher accuracy than if it were trained alone—likely due to a regularization effect.
  • Scheme-B (Teacher-Guided): A pre-trained, fixed full-precision teacher guides a low-precision student being trained from scratch. This is the most "traditional" distillation approach.
  • Scheme-C (Fine-Tuning): This is the most effective method. It starts with a trained full-precision model, lowers its precision (quantizes it), and then fine-tunes it using KD from a large teacher.

The quantization focuses on ternary precision (where weights are restricted to -1, 0, or 1) and 4-bit precision, targeting the ResNet architecture (ResNet-18, 34, and 50).

Key Insights and Ideas

  • The "Dark Knowledge" Advantage: Standard training uses "hard labels" (e.g., "this is a dog"). Distillation uses "soft labels" from the teacher, which might say "this is 90% dog, 9% cat, and 1% car." This extra information helps the low-precision student learn the underlying structure of the data more efficiently.
  • Mutual Benefit: In Scheme-A, the low-precision student acts as a form of noise or regularization for the teacher, actually improving the teacher's final accuracy.
  • Convergence Speed: Using a teacher (Scheme-B) allows low-precision models to converge 10-20% faster than training them with hard labels alone.
  • Ternary vs. Sparsity: The authors point out that ternary models are inherently sparse (about 50% of weights are zero). They argue that ternary quantization is a more efficient way to compress models than traditional pruning/sparsification, which requires complex metadata to track non-zero values.

Key Results

The Apprentice approach consistently outperformed previous state-of-the-art low-precision techniques by 1.5% to 3% in Top-1 accuracy.

Accuracy gap recovery for ResNet-18 using Apprentice
  • ResNet-18 (4-bit): As shown in the data, the 4-bit student actually outperformed the full-precision baseline in some configurations, showing that the combination of KD and quantization can be a powerful regularizer.
  • ResNet-50 (Ternary): Using Scheme-C, the authors achieved a 24.7% Top-1 error rate. For context, the full-precision baseline is 23.8%. This means they compressed the model weights by roughly 16x while losing less than 1% in accuracy.
  • CIFAR-10: The authors found that a student's accuracy improves as the teacher gets deeper, though this benefit eventually saturates.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that finding the right learning rates and balancing the loss functions ($\alpha, \beta, \gamma$ weights) requires careful tuning and lacks a "universal recipe."
  • Overkill for Small Models: The authors observed that distillation might be "overkill" for simpler architectures like AlexNet on smaller datasets, where standard quantization already performs reasonably well.
  • Future Directions: The authors suggest investigating "hyper-sparse" models and exploring whether making layers wider (rather than deeper) could further recover accuracy in ultra-low-precision settings (like 1-bit binary networks).

Summary updated

Abstract

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems -- the models (often deep networks or wide networks or both) are compute and memory intensive. Low precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low precision networks can be significantly improved by using knowledge distillation techniques. We call our approach Apprentice and show state-of-the-art accuracies using ternary precision and 4-bit precision for many variants of ResNet architecture on ImageNet dataset. We study three schemes in which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.
BibTeX
@inproceedings{mishra2018apprentice,
  author = {Asit Mishra and Debbie Marr},
  title = {{Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=B1ae1lZRb},
}
Figure