Mixed Precision DNNs: All you need is a good parametrization

Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso García, Stephen Tiedemann, Thomas Kemp, Akira Nakamura

Introduction

Training mixed-precision neural networks—where each layer has a different bitwidth—is usually a complex optimization headache. This paper demonstrates that the secret to stable, high-performance quantization isn't a complex new algorithm, but simply choosing the right way to mathematically represent (parametrize) the quantizer. By learning the step size and dynamic range instead of the bitwidth directly, researchers can train state-of-the-art mixed-precision models using standard gradient descent.

The Challenge of Quantization

To run deep learning models on mobile or embedded devices, we often rely on quantization: representing weights and activations with low-bit integers (like 4-bit or 8-bit) instead of standard 32-bit floats. Mixed-precision networks take this a step further by allowing "important" layers to use more bits and "less important" layers to use fewer, maximizing accuracy for a fixed memory budget.

However, training these models is notoriously difficult. The "rounding" function used in quantization has a derivative of zero almost everywhere, which breaks backpropagation. While researchers commonly use the Straight-Through Estimator (STE) to "fake" a gradient, learning the optimal bitwidth for each layer remains highly unstable. Previous attempts often relied on Reinforcement Learning or expensive search processes because directly optimizing the bitwidth leads to erratic training behavior.

Differentiable Quantization and Parametrization

The authors propose Differentiable Quantization (DQ) to solve this instability. The core idea is that a quantizer is defined by three interdependent variables:

  1. Step size ($d$): The distance between two discrete values.
  2. Dynamic range ($q_{\max}$): The maximum value the quantizer can represent.
  3. Bitwidth ($b$): The number of bits used, related to the others by $q_{\max} \approx 2^{b-1} \cdot d$.

The paper’s breakthrough insight is that how you pair these variables matters. They compare three parametrizations:

  • U1: Learning $(b, d)$
  • U2: Learning $(b, q_{\max})$
  • U3 (Proposed): Learning $(d, q_{\max})$ and calculating $b$ from them.
Comparison of gradient magnitudes

In U1 and U2, the gradient of the bitwidth $b$ grows exponentially because $b$ sits in the exponent. This causes "exploding gradients" and training instability. In contrast, U3 has bounded gradients and "decoupled" components, making it much easier for standard optimizers like SGD or ADAM to handle.

To enforce memory constraints (e.g., "the whole model must be under 2MB"), the authors use a penalty framework that turns the constraint into a differentiable loss term. This allows the network to automatically "trade" bits between layers during training.

Key Insights: Why U3 Succeeds

Derivatives of the quantization function
  • Parametrization is a First-Class Citizen: We often think about what we optimize, but how we represent those parameters can fundamentally change the geometry of the loss surface. U3 creates a "flatter" error surface with fewer steep ridges compared to U1 or U2.
  • Bitwidth as a Derivative: You don't need to optimize the bitwidth $b$ directly. By optimizing the physical properties of the quantizer (its resolution and its limits), the optimal bitwidth emerges naturally.
  • Diagonal Hessian: The U3 parametrization leads to a nearly diagonal Hessian matrix. In optimization terms, this means that changing the step size doesn't wildly interfere with the dynamic range, allowing for much faster and more stable convergence.
  • Mixed Precision for "Free": Because the method uses standard gradients, training a mixed-precision model takes roughly the same amount of time as training a standard 32-bit float model.

Experimental Results

The authors evaluated their approach on standard computer vision benchmarks, including CIFAR-10 (for initial proof-of-concept) and ImageNet (to demonstrate scalability).

The U3 parametrization consistently outperformed others across different architectures:

  • MobileNetV2 (ImageNet): Achieved a top-1 error of 30.26% with a total weight size of only 1.65MB (equivalent to a uniform 4-bit model). This is remarkably close to the floating-point baseline of 29.82%.
  • ResNet-18 (ImageNet): Achieved 29.92% error at a 4-bit equivalent size, outperforming previous state-of-the-art methods that didn't even quantize the first and last layers.
  • Stability: In toy experiments with Gaussian data, U3 reached the global optimum smoothly, while U1 and U2 oscillated violently or failed to converge entirely.

Limitations and Future Work

While highly effective, the approach has a few limitations:

  • The STE Approximation: The method is stable, but it still relies on the Straight-Through Estimator, which is a coarse approximation of the true gradient.
  • Hyperparameter Tuning: The penalty method requires choosing a weighting factor ($\lambda$) for the memory constraint. While the authors provide a heuristic, it still requires some manual oversight.
  • Hardware Alignment: The paper focuses on memory footprint. However, actual hardware speedups for mixed precision depend heavily on specific chip support (like flexible bit-manipulation units), which falls outside the scope of this theoretical work.

Summary updated

Abstract

Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.
BibTeX
@inproceedings{uhlich2020mixed,
  author = {Stefan Uhlich and Lukas Mauch and Fabien Cardinaux and Kazuki Yoshiyama and Javier Alonso García and Stephen Tiedemann and Thomas Kemp and Akira Nakamura},
  title = {{Mixed Precision DNNs: All you need is a good parametrization}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=B25maGxZYyQ},
}
Figure