Differentiable Quantization of Deep Neural Networks

Stefan Uhlich, Lukas Mauch, Kazuki Yoshiyama, Fabien Cardinaux, Javier Alonso García, Stephen Tiedemann, Thomas Kemp, Akira Nakamura

Introduction

Deploying Deep Neural Networks (DNNs) on edge devices requires significant compression, often achieved through quantization—reducing the precision of weights and activations from 32-bit floats to lower bitwidths. Mixed-precision networks, which assign varying bitwidths to different layers, offer a superior trade-off between accuracy and efficiency compared to uniform quantization. However, determining the optimal bitwidth for each layer is a complex combinatorial problem. While gradient-based methods using the Straight-Through Estimator (STE) are attractive for their efficiency, directly learning bitwidths often leads to unstable training and exploding gradients. This paper introduces a stable Differentiable Quantization (DQ) framework by re-parameterizing the quantizer to ensure mathematical stability during optimization.

Methods: Re-parameterizing the Quantizer

The core novelty of this work lies in the mathematical re-parameterization of the uniform quantizer. A standard quantizer relies on three interdependent variables: bitwidth ($b$), step size ($d$), and maximum dynamic range ($q_{max}$). The authors analyze three parameterization strategies:

  1. Case U1: $\theta = [b, d]^T$
  2. Case U2: $\theta = [b, q_{max}]^T$
  3. Case U3: $\theta = [d, q_{max}]^T$

Their analysis reveals that parameterizing by bitwidth ($b$) causes the maximum gradient norm to grow exponentially as the bitwidth increases, leading to training failure. In contrast, Case U3—parameterizing by step size and dynamic range—decouples the partial derivatives and ensures the gradient magnitude remains strictly bounded.

Comparison of gradient norm surfaces for different parameterizations. Cases U1 and U2 show exponential growth (exploding gradients), while Case U3 remains stable.

Key Insights: Stability and Emergent Bitwidth

The stability of the "Goldilocks" parameterization (Case U3) allows standard gradient descent to behave similarly to second-order Newton optimization. By optimizing $d$ and $q_{max}$, the optimal bitwidth emerges naturally as a secondary property, avoiding the pitfalls of direct bitwidth optimization. This stability enables the use of standard learning rates without risking divergence.

Partial derivatives of the quantization function. Case U3 (c) demonstrates bounded, stable derivatives compared to the unstable behavior in U1 and U2.

To maintain hardware constraints, the authors formulate training as a constrained optimization problem. Penalty terms are added to the loss function to enforce user-defined budgets for weight and activation memory. This approach enables the discovery of heterogeneous bitwidth distributions that consistently outperform homogeneous schemes under the same memory constraints.

Experimental Results

The proposed method was validated on CIFAR-10 and ImageNet:

  • CIFAR-10 (ResNet-20): Under a 70KB memory constraint, the model achieved an 8.58% error rate, significantly better than fixed 2-bit or 4-bit baselines.
  • ImageNet (MobileNetV2): The method successfully trained a mixed-precision MobileNetV2 with a 1.55MB weight budget. It achieved a 30.26% error rate, nearly matching the unquantized float32 baseline (29.82%) and vastly outperforming a fixed 4-bit model (36.27%).

Crucially, the training efficiency is high; training a quantized MobileNetV2 takes approximately the same time as a standard float32 model (e.g., 37 minutes per epoch on four V100 GPUs), avoiding the massive computational overhead of reinforcement learning-based search.

Conclusion and Limitations

Differentiable Quantization via stable parameterization provides a powerful tool for optimizing mixed-precision networks. While the method relies on the biased Straight-Through Estimator and requires some tuning of penalty weights ($\lambda$), it demonstrates that state-of-the-art compression can be achieved through standard gradient descent. Future work may focus on automating penalty weight selection and extending the framework to more diverse architectures like Transformers.

Summary updated

BibTeX
@inproceedings{uhlich2020differentiable,
  author = {Stefan Uhlich and Lukas Mauch and Kazuki Yoshiyama and Fabien Cardinaux and Javier Alonso García and Stephen Tiedemann and Thomas Kemp and Akira Nakamura},
  title = {{Differentiable Quantization of Deep Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=B25maGxZYyQ},
}
Figure