Neural Networks with Quantization Constraints

Ignacio Hounie, Juan Elenter, Alejandro Ribeiro

TL;DR

This paper reimagines Quantization-Aware Training (QAT) not as a search for a low-precision model, but as a constrained optimization problem where a high-precision model is forced to stay "close" to its quantized counterpart. By leveraging the mathematical property of strong duality, the authors eliminate the need for traditional gradient approximations (like STE) and use dual variables to automatically identify which layers are most sensitive to bit-reduction.

What Problem Does This Paper Address?

Deploying deep learning models on edge devices (like smartphones or IoT sensors) requires "quantization"—reducing the precision of weights and activations from 32-bit floats to much smaller formats (like 4-bit or even 1-bit).

However, training these models is notoriously difficult. The quantization function is a "step function," meaning its gradient is zero almost everywhere and undefined at the steps. To get around this, researchers usually use a "Straight-Through Estimator" (STE), which essentially "fakes" a gradient to allow backpropagation. Furthermore, not all layers in a network are equally sensitive to quantization; some can be compressed heavily, while others need higher precision to maintain accuracy. Finding this "mixed-precision" balance usually requires an exhaustive, computationally expensive search.

Methods

The authors propose Primal Dual Quantization-Aware Training (PDQAT). Instead of directly optimizing a low-precision model, they optimize a high-precision model $f$ subject to a constraint: it must remain within a certain distance $\epsilon$ of its quantized version $f^q$.

Key technical highlights include:

  • Constrained Learning: They formulate the problem as minimizing the standard loss subject to proximity constraints at every layer.
  • Strong Duality: Despite neural networks being non-convex, the authors show that this specific functional formulation has a zero duality gap. This allows them to solve the problem using a primal-dual algorithm that alternates between updating the model weights (primal) and the constraint multipliers (dual).
  • Eliminating STE: Because the optimization happens on the high-precision model while the quantization is handled via the constraint/Lagrangian, they bypass the need to estimate gradients for the quantization operation itself.
  • Sensitivity Analysis: The "dual variables" ($\lambda$) in their optimization act as a built-in sensitivity gauge. A high $\lambda$ value for a specific layer indicates that the model's performance is highly sensitive to quantization at that point.

Datasets

The researchers evaluated their approach using the CIFAR-10 image classification benchmark. They utilized a ResNet-20 architecture, a standard choice for testing efficiency-focused neural network techniques.

Key Insights and Ideas

  • The "Aha" Moment on Duality: In many optimization problems, non-convexity makes duality useless. However, by treating the problem in a functional space (Statistical Learning), the authors leverage recent results showing that these problems behave like convex ones, allowing for stable primal-dual training.
  • Dual Variables as Sensors: Usually, deciding which layers should be 8-bit vs. 4-bit is a guessing game or requires reinforcement learning. Here, the optimization process tells you. If a layer's dual variable $\lambda$ is large, that layer is "struggling" with the constraint, signaling it needs more bits or more attention.
  • Layer-wise vs. Global Constraints: The paper shows that imposing constraints at every single layer (layer-wise) leads to much better "robustness" than just constraining the final output of the model.

Key Results

  • Superior Accuracy: PDQAT consistently outperformed the popular DoReFa-net baseline across various bit-widths. For example, at 1-bit precision, PDQAT achieved 85.52% accuracy compared to the baseline's 84.91%.
  • Reduced Internal Drift: By using layer-wise constraints, the distance between high-precision and low-precision activations was reduced by an order of magnitude compared to models trained with only output constraints.
  • Effective Mixed-Precision: The authors demonstrated that by simply keeping the layers with the highest dual variables in high precision, they could recover almost all the accuracy of a full-precision model while still quantizing the majority of the network.

Limitations and Future Work

While the results on CIFAR-10 are promising, the paper does not yet demonstrate the approach on "at-scale" datasets like ImageNet or more complex tasks like object detection.

The authors suggest that future work should explore:

  1. Granularity: Moving from layer-wise constraints to even finer-grained constraints (e.g., channel-wise quantization).
  2. Real-world Hardware: Testing the performance on actual low-precision hardware rather than simulated quantization.
  3. Dynamic Bit-allocation: Using the dual variables to automatically adjust bit-widths during training rather than as a post-training analysis tool.

Summary updated

Abstract

Enabling low precision implementations of deep learning models, without considerable performance degradation, is necessary in resource and latency constrained settings. Moreover, exploiting the differences in sensitivity to quantization across layers can allow mixed precision implementations to achieve a considerably better computation performance trade-off. However, backpropagating through the quantization operation requires introducing gradient approximations, and choosing which layers to quantize is challenging for modern architectures due to the large search space. In this work, we present a constrained learning approach to quantization aware training. We formulate low precision supervised learning as a constrained optimization problem, and show that despite its non-convexity, the resulting problem is strongly dual and does away with gradient estimations. Furthermore, we show that dual variables indicate the sensitivity of the objective with respect to constraint perturbations. We demonstrate that the proposed approach exhibits competitive performance in image classification tasks, and leverage the sensitivity result to apply layer selective quantization based on the value of dual variables, leading to considerable performance improvements.
BibTeX
@inproceedings{hounie2023neural,
  author = {Ignacio Hounie and Juan Elenter and Alejandro Ribeiro},
  title = {{Neural Networks with Quantization Constraints}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094663},
}
Figure