Reducing Underflow in Mixed Precision Training by Gradient Scaling

Ruizhe Zhao, Brian Vogel, Tanvir Ahmed, Wayne LukIJCAI 2020 · DOI

TL;DR

This paper introduces "gradient scaling," a novel method to combat gradient underflow in mixed-precision training (MPT) of deep neural networks. By analytically calculating per-gradient scaling factors on-the-fly, it effectively mitigates underflow without numerical issues like overflow, leading to improved accuracy and reduced training effort compared to existing MPT techniques.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is computationally expensive. Mixed-precision training (MPT) offers a solution by using a combination of 32-bit (FP32) and 16-bit (FP16) floating-point formats. FP16, with its smaller range, can significantly speed up computations and reduce memory usage. However, a major challenge arises because gradients, which are often very small, can fall below the minimum representable value in FP16, leading to "underflow" (becoming zero). This loss of information during backpropagation hinders model accuracy and training. Existing methods like "loss scaling" (multiplying the loss by a large factor) help, but they apply a single uniform scale, which is insufficient for networks with diverse gradient distributions across layers, often requiring tedious hyperparameter tuning and potentially causing overflow.

Methods

The core contribution is gradient scaling, a method that analytically calculates and applies a specific scaling factor to each gradient tensor during backpropagation, just before it's type-casted to FP16. This is a departure from traditional loss scaling, which applies a single scale to the entire loss.

Key aspects of their approach include:

  • Statistical Model for Underflow Estimation: They develop a statistical model to estimate the underflow rate of gradients based on their distribution. This model uses assumptions about gradient independence and normal/log-normal distributions for different network components.
  • Per-Gradient Scaling: Instead of a global loss scale, gradient scaling computes an optimal scale ($\beta$) for each gradient tensor. This scale is determined by a constrained optimization problem: minimize the underflow rate while ensuring no overflow occurs.
  • On-the-Fly Calculation: The scaling factors are calculated dynamically as backpropagation progresses, adapting to the specific gradient distributions at each layer.
  • Handling Different Operators: The algorithm is designed to integrate seamlessly with various backpropagation operators, including GEMM (matrix multiplication), type casting, concatenation, and branching, by propagating and adjusting the scaling factors accordingly.

Datasets

The paper evaluates their method on standard computer vision datasets:

  • ImageNet (ILSVRC 2012): Used for evaluating ResNet-18 and ResNet-50 models.
  • PASCAL VOC 2007 and 2012: Used for evaluating SSD (Single Shot MultiBox Detector) models for object detection.
  • CamVid: Used for evaluating SegNet models for image segmentation.

Key Insights and Ideas

  • Uniform Scaling is Insufficient: The paper strongly argues that a single, uniform loss scale is suboptimal because different layers and network components can have vastly different gradient distributions. This leads to either underflow in some parts or overflow in others.
  • Per-Gradient Adaptability: Gradient scaling's ability to adapt scales to individual gradient tensors is its key advantage. This allows for finer control and better handling of diverse distributions, especially in complex architectures like SSD with branching.
  • Balancing Underflow and Overflow: The method explicitly models and constrains both underflow and overflow, ensuring numerical stability while maximizing the benefits of low precision.
  • Reduced Hyperparameter Tuning: By analytically deriving scales, gradient scaling significantly reduces the need for manual hyperparameter tuning, making MPT more accessible and easier to use.

Key Results

The proposed gradient scaling (GS) method consistently outperformed existing MPT techniques (Fixed-LS and Dyn-LS) across various tasks and architectures:

  • Image Recognition (ResNet-18/50): GS achieved the best test accuracy among FP16 methods, though the benefit was less pronounced due to the residual structure's inherent mitigation of underflow.
  • Object Detection (SSD-300/512): GS achieved comparable or better performance than the best Fixed-LS results and significantly outperformed Dyn-LS. It was noted that Fixed-LS requires extensive search for optimal scales, making GS more efficient.
  • Segmentation (SegNet): GS achieved the best MPT results, slightly below the FP32 baseline, and demonstrated more stable validation performance compared to Dyn-LS.
  • Efficiency: GS reduces the need for tedious hyperparameter tuning required by Fixed-LS, making MPT more practical.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that for extremely small pixel-wise softmax gradients in segmentation tasks, achieving full mitigation might require scales exceeding the maximum representable value, leading to a gap with the FP32 baseline.
  • While the statistical model is derived, it relies on certain assumptions about gradient distributions.
  • Future Work:
    • Evaluate gradient scaling on Natural Language Processing (NLP) tasks and models.
  • Extend the method to even lower-precision representations like 8-bit floating-point and fixed-point formats.

Summary updated

BibTeX
@inproceedings{zhao2020reducing,
  author = {Ruizhe Zhao and Brian Vogel and Tanvir Ahmed and Wayne Luk},
  title = {{Reducing Underflow in Mixed Precision Training by Gradient Scaling}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/404},
}
Figure