In-Hindsight Quantization Range Estimation for Quantized Training

Marios Fournarakis, Markus Nagel

Abstract: Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing back-propagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet).

The Challenge of Fully Quantized Training

While 8-bit precision inference is now standard for mobile and edge devices, training models entirely in low-bit precision remains a significant hurdle. The primary difficulty lies in the backward pass. Unlike weights, which remain relatively stable, gradients are unbounded and their distributions shift dynamically throughout the training process.

To accommodate these shifts, researchers typically rely on dynamic quantization. This process requires scanning an entire gradient tensor to find its minimum and maximum values before scaling it into 8 bits. However, modern hardware accelerators process data in small slices. To find the global min/max of a tensor, the hardware must write the full-precision tensor to memory, scan it, and then read it back to perform quantization. This "memory wall" creates massive overhead, often negating the computational speedups gained from low-bit arithmetic.

Comparison of static and dynamic quantization flows, illustrating the memory round-trip required for dynamic range estimation.

Proposed Method: In-Hindsight Range Estimation

To bypass the memory bottleneck of dynamic quantization, we introduce In-Hindsight Min-Max. This hardware-friendly alternative treats quantization as a static operation by leveraging the temporal smoothness of gradient distributions.

Instead of calculating the range for the current tensor $t$ in a blocking fashion, the system uses the range estimated from the previous iteration ($t-1$). The process follows three steps:

  1. Quantize with the Past: The current tensor is quantized using a pre-stored range as it is generated, enabling fast, single-pass processing.
  2. Update for the Future: Simultaneously, the hardware tracks the min/max statistics of the current tensor "online" at the accumulator level.
  3. Exponential Moving Average (EMA): At the end of the step, these new statistics update the stored range for the next iteration ($t+1$) using an EMA formula.
The In-Hindsight update mechanism: using the current range $q^t$ for quantization while calculating $q^{t+1}$ for the next step.

By using "hindsight," the system avoids storing full-precision tensors in memory solely for scaling factor calculations, effectively turning a dynamic problem into a static one with minimal hardware overhead.

Experimental Results and Efficiency

We evaluated the In-Hindsight method on standard image classification benchmarks, including Tiny ImageNet and the full ImageNet dataset, across architectures like ResNet18 and MobileNetV2.

Accuracy and Regularization

Our experiments show that In-Hindsight estimation achieves accuracy parity with full-precision (FP32) baselines. On Tiny ImageNet with ResNet18, the method reached 59.46% accuracy, slightly exceeding the FP32 baseline of 58.97%. This suggests that the noise introduced by 8-bit quantization may act as a form of regularization, potentially improving model generalization.

Hardware Efficiency

The most significant gain is in memory traffic. By eliminating the need for dynamic range scanning, the method reduces memory movement by up to 8x for typical deep learning layers. For specific layers, such as 1x1 convolutions in MobileNetV2, dynamic quantization was found to require 685% more memory movement than our proposed static approach.

Conclusion

In-Hindsight Range Estimation provides a simple, drop-in replacement for dynamic quantization in neural network training. By decoupling range estimation from the immediate quantization step, it resolves the memory bottleneck that has previously hindered the efficiency of 8-bit training. While it requires a brief initial calibration phase, its ability to maintain accuracy while drastically reducing memory traffic makes it a compelling choice for future on-device learning hardware.

Summary updated

Abstract

Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing back-propagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet)
BibTeX
@inproceedings{fournarakis2021hindsight,
  author = {Marios Fournarakis and Markus Nagel},
  title = {{In-Hindsight Quantization Range Estimation for Quantized Training}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=VhHhHoOkN4B},
}
Figure