Training Neural Networks with Low-Precision Model Memory

Bingrui Li, Ziteng Wang, Jianfei Chen, Jun ZhuICLR 2023 · OpenReview

TL;DR

This paper introduces Low-Precision Model Memory (LPMM), a novel framework that drastically reduces the memory footprint of training large neural networks by quantizing not just momentum, but also model parameters and gradient accumulators to low precision. They tackle the critical "arithmetic underflow" problem with stochastic quantization and microbatching, achieving over 70% memory reduction with negligible accuracy loss on standard benchmarks.

What Problem Does This Paper Address?

Training massive neural networks, like those powering modern AI, is severely hampered by memory limitations. A significant portion of this "model memory" is dedicated to storing parameters, gradients, and optimizer states (like momentum). While previous methods focused on compressing only the momentum, this paper addresses the inefficiency of not quantizing the entire model memory. The core challenge they tackle is arithmetic underflow: when low-precision numbers are so small that adding them to larger numbers effectively results in zero, leading to stalled training or incorrect updates.

Methods

The paper proposes Low-Precision Model Memory (LPMM), an optimization framework that quantizes all model-related statistics:

  • Model Parameters: Stored in low precision (e.g., 12-bit).
  • Gradients: Quantized for accumulation (e.g., 8-bit).
  • Momentum: Quantized (e.g., 8-bit).

To combat the identified arithmetic underflow problem, LPMM introduces two key techniques:

  • Unbiased Stochastic Quantization: Instead of deterministic rounding, they use stochastic rounding. This probabilistic approach ensures that, on average, the quantized values are unbiased, preventing systematic errors and allowing small updates to eventually contribute.
  • Microbatching: This technique involves accumulating gradients from multiple mini-batches before applying the update to the momentum and parameters. This effectively "amplifies" the update signal, making it less susceptible to being lost due to low precision.

Datasets

The paper evaluates LPMM on the ImageNet dataset for image classification tasks using ResNet-18 and ResNet-50 models. They also report experiments on the GLUE benchmark for natural language understanding tasks using the RoBERTa-Large model.

Key Insights and Ideas

  • Underflow is the Bottleneck: The paper clearly identifies arithmetic underflow as the primary obstacle to achieving significant memory savings with low-precision optimizers, rather than just quantization error itself.
  • Stochasticity is Key for Convergence: Deterministic quantization can lead to training stalls. Unbiased stochastic quantization is crucial for maintaining convergence guarantees, especially when updates become small.
  • Microbatching as an Amplifier: Microbatching effectively increases the magnitude of updates, making them more robust to low-precision representation and mitigating underflow. It's presented as a way to trade off gradient quantization error for reduced parameter quantization error.
  • Holistic Memory Compression: The true power of LPMM lies in its comprehensive approach to quantizing all model memory components, not just one. This unified framework unlocks substantial memory savings.

Key Results

  • Significant Memory Reduction: LPMM-SGDM reduces model memory by over 70% compared to full-precision SGDM. For example, they achieve 12-bit parameters, 8-bit gradients, and 8-bit momentum, requiring only 28 bits per parameter, compared to 72-96 bits for full-precision.
  • Negligible Accuracy Loss: On ImageNet, LPMM-SGDM achieves comparable accuracy to full-precision SGDM (e.g., 71.0% vs. 71.1% for ResNet-18, and 77.3% vs. 77.2% for ResNet-50).
  • Transformer Performance: LPMM-Adam also demonstrates strong performance on RoBERTa-Large for GLUE tasks, achieving near-lossless accuracy with 8-bit quantization for gradients, momentum, and square momentum, and 12-bit for parameters.
  • Effectiveness of Techniques: Ablation studies confirm that both stochastic quantization and microbatching are critical for LPMM's success, especially at lower bitwidths (8-bit).

Limitations and Future Work

  • Convexity Assumption: The theoretical convergence analysis is performed under the assumption of a convex objective function, which is a simplification. The authors acknowledge that extending this to non-convex settings is a future research direction.
  • Implementation Overhead: While LPMM offers significant memory savings, the paper notes that current implementations might incur some computational overhead due to the need for dequantization and re-quantization. They suggest that a fully CUDA-implemented framework could mitigate this.
  • Dynamic Quantization: The paper mentions using dynamic quantization to balance overflow and underflow, but a deeper exploration of optimal dynamic strategies could be a future avenue.
  • Non-linear Quantizers: While the analysis is applicable to non-linear quantizers, the primary focus is on symmetric quantization. Further investigation into the behavior of LPMM with more complex non-linear quantization schemes could be beneficial.

Summary updated

Abstract

The demand for memory to store model-related statistics ("model memory") is a major bottleneck for training large neural networks. A promising solution is low-precision optimizers, which reduce the numerical precision of the model memory. However, existing work only compresses the momentum, resulting in suboptimal memory efficiency. This paper proposes Low-Precision Model Memory (LPMM), an optimization framework with the entire model memory kept in low precision. LPMM compresses not only the momentum but also model parameters and gradient accumulators. We identify arithmetic underflow as the main problem in building low-precision optimizers and propose a stochastic quantization method and a microbatching technique to overcome this problem. We analyze the convergence behavior of LPMM and theoretically show how the proposed techniques could affect underflowing, which in turn affects the convergence. We apply LPMM to the SGD optimizer with momentum (SGDM). On several realistic benchmarks, LPMM-SGDM can train neural networks with negligible loss of accuracy while reducing over 70% of the model memory compared to the full-precision SGDM.
BibTeX
@inproceedings{li2023training,
  author = {Bingrui Li and Ziteng Wang and Jianfei Chen and Jun Zhu},
  title = {{Training Neural Networks with Low-Precision Model Memory}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=cs3n00FQ7OI},
}
Figure