Abstract
The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by randomly changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.
Introduction
Quantizing deep neural networks (QDNNs)—reducing the bit-width of weights and activations—is essential for deploying AI on resource-constrained edge devices. However, quantization usually leads to a significant drop in accuracy. To recover this loss, researchers often use Knowledge Distillation (KD), where a high-performance "teacher" model guides a smaller "student" model. While effective, standard KD requires a large, memory-intensive teacher model to be present during training, making on-device model adaptation difficult. SPEQ addresses this by introducing a self-distillation framework where the teacher and student share the exact same model parameters.
The SPEQ Framework
The core innovation of SPEQ is the use of two parallel forward-pass paths that share weights but differ in activation precision.
- Target Precision Path (TPP): This path represents the student model running at the target low precision (e.g., 2-bit).
- Stochastic Precision Path (SPP): To create a "teacher" signal, the bit-precision of the activations for each layer is chosen randomly (e.g., a 50/50 chance of being 2-bit or 8-bit) at each iteration.
By varying the precision every iteration, the student is essentially learning from an "ensemble" of different quantization configurations. This allows the model to learn how to filter out the noise introduced by low-precision activation quantization.
Robust Self-Distillation
Standard KD uses KL-divergence, which forces the student to match the teacher's distribution exactly. However, in self-distillation, the teacher isn't always "smarter" than the student. SPEQ employs Cosine Similarity Loss instead. This choice is more robust; if the teacher's output is ambiguous or noisy, the gradient effectively diminishes, allowing the student to rely more on the ground-truth label rather than following a potentially incorrect teacher signal.
Key Insights and Results
The researchers observed that while weight quantization shifts decision boundaries, activation quantization primarily adds noise to those boundaries. By stochastically varying activation precision, SPEQ "samples" this noise and learns to minimize its impact.
SPEQ demonstrated state-of-the-art performance across several benchmarks:
- Image Classification: On CIFAR-10, a 2-bit ResNet20 trained with SPEQ achieved 91.4% accuracy, outperforming standard retraining (90.7%) and even beating KD methods that used a much larger, full-precision teacher (91.2%).
- BERT Compression: On the SQuAD dataset, SPEQ significantly closed the gap between fixed-point BERT and its floating-point counterpart in 3-bit and 4-bit configurations.
- Transfer Learning: In low-data scenarios (Oxford Flowers-102), a 2-bit ResNet trained with SPEQ outperformed a full-precision model trained with standard cross-entropy, demonstrating a powerful regularizing effect.
Conclusion
SPEQ provides a highly efficient way to boost the accuracy of quantized networks without the overhead of a separate teacher model. By leveraging stochastic precision and robust cosine similarity loss, it enables high-performance QDNNs suitable for edge deployment and on-device training.
Summary updated