Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

Yoonho Boo, Sungho Shin, Jungwook Choi, Wonyong Sung

Abstract

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.

Introduction

Quantization is a critical technique for deploying deep neural networks on resource-constrained edge devices. By reducing the precision of weights and activations from 32-bit floating-point to low-bit integers, significant gains in memory and power efficiency are achieved. However, this reduction often leads to accuracy degradation. Knowledge Distillation (KD) has emerged as a powerful remedy, where a high-precision "teacher" model guides a low-precision "student." Despite its efficacy, traditional KD requires maintaining a separate, often large, teacher model, which is impractical for on-device fine-tuning or scenarios with limited memory.

Methodology: Stochastic Precision Ensemble (SPEQ)

SPEQ introduces a self-distillation framework where the teacher and student share the same model parameters, eliminating the need for an external teacher. The training process involves two parallel forward passes: the Target Precision Path (TPP) and the Stochastic Precision Path (SPP).

The TPP represents the student, operating at the final target low-bit precision. Conversely, the SPP acts as the teacher. In the SPP, the activation precision for each layer is stochastically assigned between a low-bit (e.g., 2-bit) and a higher-bit (e.g., 8-bit) value. This stochasticity ensures that the teacher provides a diverse set of "soft labels" across training iterations, effectively creating a stochastic ensemble.

The SPEQ training structure showing the Target Precision Path (student) and Stochastic Precision Path (teacher) sharing the same model layers.

A key component of SPEQ is the use of cosine similarity loss for distillation. Unlike the standard KL-divergence, cosine similarity allows the student to be "confidence-aware." If the teacher's prediction is noisy or unconfident (resulting in a flatter distribution), the gradient signal is naturally attenuated, preventing the student from learning from unreliable signals.

Insights and Experimental Results

The core intuition behind SPEQ is that activation quantization primarily acts as noise. By varying the precision in the SPP, the model learns to identify and ignore this noise, refining its decision boundaries. The diversity of the teacher guidance is evident in the varying softmax distributions produced by different stochastic paths for the same input.

Left (a): The ratio of 8-bit selection during training. Right (b): Softmax distributions generated by the stochastic path, illustrating teacher diversity.

SPEQ was evaluated across multiple domains:

  • Image Classification: On ImageNet, SPEQ improved the top-1 accuracy of a 2-bit ResNet18 by approximately 1%, significantly closing the gap with the full-precision baseline. On CIFAR-10, a 2-bit ResNet20 achieved 91.4% accuracy, surpassing other KD-based quantization methods.
  • Natural Language Processing: For a 3-bit quantized BERT model on the SQuAD 1.1 dataset, SPEQ boosted the F1 score from 81.4 to 85.1, demonstrating its applicability beyond vision tasks.
  • Transfer Learning: SPEQ showed remarkable robustness in low-data regimes. In transfer learning tasks like Oxford Flowers-102, it outperformed even full-precision models when training samples were extremely limited (e.g., 5 samples per class).

Conclusion

SPEQ provides a lightweight and effective solution for training high-performance quantized networks. By leveraging self-distillation through stochastic precision paths and cosine similarity loss, it achieves state-of-the-art results without the overhead of a separate teacher model. This makes it particularly suitable for on-device learning and deployment in memory-constrained environments. Future work may explore extending stochasticity to weight quantization to further enhance model robustness.

Summary updated

Abstract

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.
BibTeX
@inproceedings{boo2021stochastic,
  author = {Yoonho Boo and Sungho Shin and Jungwook Choi and Wonyong Sung},
  title = {{Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=rZ448U-FSxc},
}
Figure