In-Hindsight Quantization Range Estimation for Quantized Training

Marios Fournarakis, Markus Nagel

Abstract

Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing back-propagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet).

Introduction: The Bottleneck of Dynamic Quantization

While quantizing neural networks for inference is a well-established practice, fully quantized training—where both forward and backward passes are quantized—remains a significant challenge. The primary hurdle lies in the nature of gradients: they are unbounded and their distributions shift significantly throughout the training process.

To prevent catastrophic quantization error, existing methods rely on dynamic quantization. This approach calculates the min/max range of a tensor on the fly to define the quantization grid. However, in fixed-point hardware accelerators, this creates a massive memory bottleneck. Because the quantization range depends on the entire tensor, the full 32-bit output of an accumulator must be written to memory, analyzed for its range, read back for quantization, and then written to memory again in its low-bit format. This redundant data movement can increase memory transfer by up to 8×, severely slowing down training and wasting power on resource-constrained edge devices.

Proposed Method: In-Hindsight Range Estimation

To eliminate the overhead of dynamic quantization, we propose in-hindsight range estimation. This method shifts the paradigm from calculating the range of the current tensor to leveraging historical data.

Static Quantization with Historical Ranges

Instead of a pre-quantization pass, the method uses an exponential moving average (EMA) of min-max statistics from previous iterations. By using the quantization parameters pre-computed at step $t-1$, the hardware can perform static quantization on the accumulator's output immediately at step $t$. This allows the system to write only the low-bit representation to memory, bypassing the need to store high-precision intermediate tensors.

Comparison of static (in-hindsight) and dynamic quantization flows for gradients.

Online Statistic Extraction

While the output is being computed and quantized, simple hardware logic tracks the new min/max statistics directly from the accumulator. Once the layer operation is complete, these statistics are used to update the EMA for the next iteration. This "in-hindsight" approach ensures that the quantization grid evolves with the training process without requiring additional memory passes.

Experimental Results

We evaluated the effectiveness of in-hindsight range estimation on standard image classification benchmarks, including Tiny ImageNet and ImageNet, using architectures like ResNet18 and MobileNetV2.

Memory Efficiency

The primary advantage of our method is the drastic reduction in memory bandwidth. Our analysis shows that dynamic quantization incurs a +685% memory overhead for MobileNetV2 1×1 convolutions and +366% for ResNet18 3×3 convolutions. In-hindsight estimation completely avoids this overhead by enabling a single-pass write of quantized values.

Memory traffic analysis comparing static (in-hindsight) and dynamic quantization for activations.

Accuracy Parity

Despite using "stale" ranges from previous iterations, the method maintains accuracy comparable to full-precision (FP32) training:

  • Tiny ImageNet: An 8-bit ResNet18 using in-hindsight min-max achieved 58.99% accuracy, slightly exceeding the FP32 baseline of 58.97%.
  • ImageNet: The 8-bit ResNet18 reached 69.37% accuracy, nearly matching the FP32 baseline of 69.75% and outperforming standard dynamic min-max methods (69.21%).

Conclusion

In-hindsight range estimation demonstrates that historical statistics are a reliable predictor of current tensor distributions in neural network training. By leveraging this insight, we can achieve the benefits of static quantization—namely, massive reductions in memory bandwidth—without the accuracy degradation typically associated with fixed ranges. This method serves as a highly efficient, hardware-friendly drop-in replacement for existing range estimation techniques.

Summary updated

Abstract

Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing back-propagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet).
BibTeX
@inproceedings{fournarakis2021hindsight,
  author = {Marios Fournarakis and Markus Nagel},
  title = {{In-Hindsight Quantization Range Estimation for Quantized Training}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=VhHhHoOkN4B},
}
Figure