Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers

Zidong Du (杜子东)arXiv 2019 · arXiv

TL;DR

This paper introduces a novel "adaptive precision training" method that dynamically adjusts the bit-width of fixed-point numbers used during neural network training. By intelligently adapting precision layer-by-layer, the approach achieves comparable accuracy to full-precision training while significantly accelerating the training process and enabling direct deployment of quantized models.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are computationally intensive, hindering their deployment on resource-constrained devices. While quantizing DNNs to use low-bit-width fixed-point numbers (e.g., int8) is effective for inference, applying this directly to training often leads to significant accuracy degradation. This is because the dramatic changes in data distribution during training, especially in backpropagation, are sensitive to precision. Existing methods either use a fixed, often higher, bit-width for all layers, or struggle to find optimal precision settings. This paper tackles the challenge of enabling efficient and accurate training of DNNs using low-bit-width fixed-point numbers for both forward and backward passes, without sacrificing accuracy.

Methods

The core of this work is the Adaptive Precision Training approach. It introduces two key components:

  • Quantization Error Measurement (QEM): This component measures the change in data distribution when quantized. It uses the difference in the mean of data before and after quantization as an indicator of how much the distribution has been altered. A larger difference suggests that the current bit-width is insufficient.
  • Quantification Parameter Adjustment (QPA): Based on the QEM's output, QPA dynamically adjusts the quantization parameters (specifically, the bit-width and resolution) for each layer. The goal is to find the smallest bit-width that doesn't significantly alter the data distribution, thus preserving accuracy. This adjustment is done layer-wise and adaptively throughout training.

The algorithm quantifies weights, activations, and activation gradients. Crucially, it applies this adaptive quantization to both the forward pass (FPROP) and the backward pass (BPROP/WTGRAD), which is a key differentiator from many prior works that only quantized the forward pass or used fixed bit-widths.

Datasets

The paper evaluates its approach on a variety of standard datasets and tasks:

  • Image Classification: ImageNet
  • Object Detection: COCO, VOC, IMG Detection
  • Segmentation: VOC
  • Machine Translation: WMT'17 news translation dataset, WMT'16 Multi30k dataset

Key Insights and Ideas

  • Data Distribution is Key: The paper's central insight is that the required precision for a layer is directly related to its data distribution. Layers with wider variance or more centralized distributions require higher precision to avoid accuracy loss.
  • Layer-wise Adaptation is Crucial: A uniform bit-width across all layers is suboptimal. Dynamically adapting precision for each layer based on its specific data characteristics is essential for maintaining accuracy.
  • Backpropagation is Sensitive: Quantizing the backward pass is more challenging than the forward pass, as gradients can have wider ranges and more complex distributions. The proposed adaptive method successfully addresses this.
  • Fixed-point Training Can Be Fast: By leveraging low-bit-width operations, especially on hardware with specialized instructions (like Intel CPUs with AVX2), significant training speedups are achievable.

Key Results

  • Accuracy Preservation: The proposed adaptive precision training achieves comparable accuracy to float32 baselines across various tasks (classification, object detection, segmentation, machine translation), with negligible accuracy losses (average -1.40%~1.3%).
  • Training Speedup: The method accelerates training by up to 2.52 times on an Intel CPU and 2.89 times on a GPU compared to float32 training.
  • Reduced Computation: The overhead for QEM and QPA is minimal, with quantization computation being within 1% for most networks.
  • Direct Deployment: The trained models can be directly deployed using their quantized weights without further fine-tuning.
  • Adaptive Nature: The paper demonstrates that for some layers (e.g., convolutional layers), int8 is sufficient, while others (e.g., fully connected layers) require higher precision (int12 or int16) to maintain accuracy.

Limitations and Future Work

  • Hardware Dependency: The speedup is dependent on hardware support for low-bit-width operations. The paper notes that for the Xeon Gold 6154, int16 x int8 multiplication was implemented as int16 x int16, suggesting potential for even higher speedups with future hardware.
  • Future Directions: The authors suggest extending their error measurement to low-bit inference (binary/ternary) and exploring gradient compression techniques. They also highlight the potential for further acceleration with flexible bit-width arithmetic operations on future hardware.

Summary updated

Abstract

Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers. Recent emerged quantization technique has been applied to inference of deep neural networks for fast and efficient execution. However, directly applying quantization in training can cause significant accuracy loss, thus remaining an open challenge.
BibTeX
@misc{2019adaptive,
  author = {Zidong Du (杜子东)},
  title = {{Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers}},
  year = {2019},
  eprint = {1911.00361},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.00361},
}
Figure