Introduction
Deploying Deep Neural Networks (DNNs) on edge devices requires significant compression, often achieved through quantization—reducing the precision of weights and activations from 32-bit floats to lower bitwidths. Mixed-precision networks, which assign varying bitwidths to different layers, offer a superior trade-off between accuracy and efficiency compared to uniform quantization. However, determining the optimal bitwidth for each layer is a complex combinatorial problem. While gradient-based methods using the Straight-Through Estimator (STE) are attractive for their efficiency, directly learning bitwidths often leads to unstable training and exploding gradients. This paper introduces a stable Differentiable Quantization (DQ) framework by re-parameterizing the quantizer to ensure mathematical stability during optimization.
Methods: Re-parameterizing the Quantizer
The core novelty of this work lies in the mathematical re-parameterization of the uniform quantizer. A standard quantizer relies on three interdependent variables: bitwidth ($b$), step size ($d$), and maximum dynamic range ($q_{max}$). The authors analyze three parameterization strategies:
- Case U1: $\theta = [b, d]^T$
- Case U2: $\theta = [b, q_{max}]^T$
- Case U3: $\theta = [d, q_{max}]^T$
Their analysis reveals that parameterizing by bitwidth ($b$) causes the maximum gradient norm to grow exponentially as the bitwidth increases, leading to training failure. In contrast, Case U3—parameterizing by step size and dynamic range—decouples the partial derivatives and ensures the gradient magnitude remains strictly bounded.
Key Insights: Stability and Emergent Bitwidth
The stability of the "Goldilocks" parameterization (Case U3) allows standard gradient descent to behave similarly to second-order Newton optimization. By optimizing $d$ and $q_{max}$, the optimal bitwidth emerges naturally as a secondary property, avoiding the pitfalls of direct bitwidth optimization. This stability enables the use of standard learning rates without risking divergence.
To maintain hardware constraints, the authors formulate training as a constrained optimization problem. Penalty terms are added to the loss function to enforce user-defined budgets for weight and activation memory. This approach enables the discovery of heterogeneous bitwidth distributions that consistently outperform homogeneous schemes under the same memory constraints.
Experimental Results
The proposed method was validated on CIFAR-10 and ImageNet:
- CIFAR-10 (ResNet-20): Under a 70KB memory constraint, the model achieved an 8.58% error rate, significantly better than fixed 2-bit or 4-bit baselines.
- ImageNet (MobileNetV2): The method successfully trained a mixed-precision MobileNetV2 with a 1.55MB weight budget. It achieved a 30.26% error rate, nearly matching the unquantized float32 baseline (29.82%) and vastly outperforming a fixed 4-bit model (36.27%).
Crucially, the training efficiency is high; training a quantized MobileNetV2 takes approximately the same time as a standard float32 model (e.g., 37 minutes per epoch on four V100 GPUs), avoiding the massive computational overhead of reinforcement learning-based search.
Conclusion and Limitations
Differentiable Quantization via stable parameterization provides a powerful tool for optimizing mixed-precision networks. While the method relies on the biased Straight-Through Estimator and requires some tuning of penalty weights ($\lambda$), it demonstrates that state-of-the-art compression can be achieved through standard gradient descent. Future work may focus on automating penalty weight selection and extending the framework to more diverse architectures like Transformers.
Summary updated