HyperBlock Floating Point: Generalised Quantization Scheme for Gradient and Inference Computation

Marcelo Gennari Do Nascimento, Victor Adrian Prisacariu, Roger Fawcett, Martin LanghammerWACV 2023 · DOI

Introduction

Training deep neural networks (DNNs) is a computationally intensive and energy-demanding process. While quantization techniques have successfully optimized models for inference on edge devices, applying low-precision arithmetic to the training phase remains a significant challenge. The primary bottleneck lies in the "backward pass," where gradients—which exhibit a much wider dynamic range than weights—must be calculated with high precision to ensure convergence.

Researchers from the University of Oxford and Intel Corporation have introduced HyperBlock Floating Point (HBFP), a novel quantization scheme designed to bring low-precision efficiency to the entire model lifecycle. By rethinking how exponents are shared across data tensors, HBFP enables efficient 4-bit training and inference with minimal impact on accuracy.

The HyperBlock Floating Point (HBFP) Scheme

Standard Block Floating Point (BFP) formats share a single exponent across a group of numbers (e.g., a single channel). However, BFP often fails during backpropagation because the process involves tensor transpositions. When a tensor is transposed, the shared exponents no longer align with the hardware's computational blocks, necessitating frequent and costly re-quantization.

HBFP addresses this by sharing exponents across a 2D block that spans both the batch and channel dimensions. This multidimensional sharing makes the format "transposition invariant."

Comparison of BFP and HBFP under transposition. HBFP maintains shared exponents across channels after transposing, unlike standard BFP.

As shown in the figure above, HBFP ensures that shared exponents remain valid even after the mathematical transformations required for backpropagation. This architectural shift allows the hardware to reuse quantized tensors, reducing the number of quantization operations per layer from six to three.

Convolution layer data flow for BFP (a) and HBFP (b). HBFP reduces the number of quantization steps required during training.

In addition to 2D block sharing, HBFP utilizes stochastic rounding. Unlike standard rounding, which can cause small gradients to vanish to zero, stochastic rounding preserves the expected value of the gradient, preventing training stagnation in low-precision regimes.

Experimental Evaluation

The authors evaluated HBFP across several standard computer vision benchmarks and architectures, including CIFAR10, SVHN, and ImageNet (ILSVRC2012), using models such as AlexNet, VGG16, and ResNet18.

Key Results

  • Accuracy Preservation: On the large-scale ImageNet dataset, training AlexNet with 4-bit HBFP (weights, activations, and gradients) resulted in no loss of accuracy compared to the FP32 baseline. For ResNet18, 4-bit HBFP achieved 69.2% top-1 accuracy, only 0.5% lower than the full-precision model.
  • Hardware Efficiency: Logic synthesis for a 10nm FinFET process indicates that an INT4 BFP multiplier is approximately 37x smaller than an FP32 multiplier and 2x more efficient than the emerging FP8 standard.
  • Ablation Studies: The importance of stochastic rounding was highlighted in tests with ResNet18; without it, the accuracy of a 5-bit model dropped from 68.3% to 40.7%.

Discussion and Future Work

HBFP demonstrates that 4-bit precision is a viable target for full-scale DNN training. By making the data format resilient to transposition, HBFP significantly slashes memory bandwidth and hardware area requirements.

However, challenges remain at extreme bitwidths. Pushing the format to 2 bits leads to significant accuracy degradation as block sizes increase. Furthermore, while HBFP is highly effective for Convolutional Neural Networks (CNNs), its application to Transformer architectures remains an area for future investigation. Subsequent research should also focus on physical hardware implementations in AI accelerators or FPGAs to validate simulated performance gains.

Summary updated

BibTeX
@inproceedings{nascimento2023hyperblock,
  author = {Marcelo Gennari Do Nascimento and Victor Adrian Prisacariu and Roger Fawcett and Martin Langhammer},
  title = {{HyperBlock Floating Point: Generalised Quantization Scheme for Gradient and Inference Computation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00630},
}
Figure