8-Bit Approximations for Parallelism in Deep Learning Tim Dettmers

Training deep neural networks on massive datasets requires parallelization across many processors and machines. However, the sequential nature of backpropagation creates a massive bottleneck: GPUs often spend more time waiting for data to transfer over the network (PCIe or InfiniBand) than they do actually computing.

Standard deep learning uses 32-bit floating-point numbers. While precise, these are "heavy" to move. Previous attempts to compress this data (like 1-bit quantization) often struggled with convergence or required very large batch sizes that slowed down learning. The challenge is to find a "sweet spot"—a data format small enough to speed up transfers but expressive enough to maintain the mathematical integrity of the training process.

This paper proposes using 8-bit approximations for gradients and activations to overcome these communication bottlenecks. By compressing data from 32-bit to 8-bit and using a clever sub-batching strategy, the author achieves up to a 50x speedup on large clusters with negligible loss in predictive accuracy.

Methods and 8-Bit Data Types

The author introduces several 8-bit data types designed to replace 32-bit floats during the communication phase:

Bit layouts for 8-bit data types
  • Dynamic Tree (The Star Performer): This format uses a dynamic exponent (0 to 7 bits) and a binary bisection tree for the mantissa. This allows the format to represent a very wide range of values—crucial for gradients, which can vary by orders of magnitude—while maintaining higher precision for smaller values.
  • Linear Quantization: This involves normalizing a matrix by its absolute maximum value and then using a 7-bit bisection tree to represent the range [0, 1].
  • Sub-batching Strategy: To maximize efficiency, the paper employs a scheme where a mini-batch is split into $K$ sub-batches. This allows the system to "hide" the communication of one sub-batch behind the computation of the next, ensuring the GPUs are rarely idle.
  • Fast Implementation: Compression and decompression are implemented using binary searches on sorted tables in GPU shared memory, taking only about 1 nanosecond per number.

Key Insights

  • Range Matters More Than Precision: The success of the "Dynamic Tree" format suggests that for deep learning gradients, being able to represent a wide range of values (via dynamic exponents) is more important than having high-bit precision for any single value.
  • Compression for Model Parallelism: While most research focuses on compressing gradients (data parallelism), this paper shows that 8-bit approximations are highly effective for compressing activations in model parallelism, setting a new state-of-the-art for that domain.
  • Solving the Large Batch Problem: Techniques like 1-bit quantization often require massive batch sizes to converge, which can actually lead to poorer final models. 8-bit approximations allow for smaller, more effective batch sizes while still providing significant communication speedups.

Key Results

The techniques were evaluated on standard benchmarks including MNIST, CIFAR-10, and ImageNet (using AlexNet) to demonstrate scalability on large-scale, real-world tasks.

  • Massive Scaling: On a simulated system of 96 GPUs, 8-bit approximations achieved a 50x speedup, compared to only 23x for standard 32-bit parallelism.
  • Near-Zero Accuracy Loss: On ImageNet, the 8-bit model achieved a Top-5 test error of 18.65%, nearly identical to the 32-bit baseline of 18.55%.
  • Bandwidth Efficiency: The method provides a consistent 2x data transfer speedup relative to 32-bit parallelism across various hardware configurations.
Top-5 classification error over epochs for 8-bit vs 32-bit

Limitations and Future Work

  • Network Latency: While 8-bit compression solves the bandwidth problem, network latency remains a significant bottleneck as clusters grow beyond a dozen nodes. The author notes that latency increases exponentially in large InfiniBand systems.
  • Stochastic Rounding: The paper notes that "stochastic rounding"—a technique where numbers are rounded up or down probabilistically—was not tested here but could potentially improve 8-bit performance even further.
  • Hardware Evolution: The author acknowledges that future hardware (like 3D GPU memory or faster interconnects) will shift the bottlenecks again, potentially requiring new algorithmic adjustments.

Summary updated

Abstract

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.
BibTeX
@inproceedings{2016bit,
  title = {{8-Bit Approximations for Parallelism in Deep Learning Tim Dettmers}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.04561},
}
Figure