Neural Networks with Quantization Constraints

Ignacio Hounie, Juan Elenter, Alejandro Ribeiro

Introduction

Deploying deep learning models on edge devices, such as smartphones or IoT sensors, necessitates quantization—the process of reducing the precision of weights and activations from 32-bit floats to lower-bit formats (e.g., 4-bit or 1-bit). While essential for efficiency, training quantized models is notoriously difficult. The quantization function is a non-differentiable step function, rendering standard gradient-based optimization ineffective. Traditionally, researchers rely on the "Straight-Through Estimator" (STE) to approximate gradients, but this heuristic lacks theoretical grounding. Furthermore, determining which layers are most sensitive to bit-reduction—a prerequisite for effective mixed-precision deployment—often requires exhaustive and computationally expensive searches.

This paper introduces Primal Dual Quantization-Aware Training (PDQAT), a novel framework that reimagines Quantization-Aware Training (QAT) as a constrained optimization problem. Instead of searching for a low-precision model directly, the authors optimize a high-precision model while enforcing constraints that keep it "close" to its quantized counterpart.

Methodology: PDQAT

The core idea of PDQAT is to minimize a standard loss function subject to proximity constraints at every layer. Specifically, the high-precision model $f$ is constrained to remain within a distance $\epsilon$ of its quantized version $f^q$.

Schematic of a residual block with quantization for forward pass (QF) and constraint evaluation (QC).

Key technical innovations include:

  1. Functional Space Optimization: By formulating the problem in a functional space (Statistical Learning), the authors leverage recent mathematical results showing that these non-convex neural network problems can exhibit zero duality gap. This allows the use of stable primal-dual algorithms.
  2. Eliminating STE: The optimization updates the high-precision model weights (primal variables) while the quantization is handled via Lagrangian multipliers (dual variables). This bypasses the need for gradient approximations like STE.
  3. Dual Variables as Sensitivity Gauges: The dual variables ($\lambda$) associated with each layer's constraint act as a built-in sensitivity measure. A high $\lambda$ value indicates that the model's performance is highly sensitive to quantization at that specific layer.

Key Insights and Results

The researchers evaluated PDQAT on the CIFAR-10 benchmark using a ResNet-20 architecture. The results highlight several advantages of the primal-dual approach.

Layer-wise vs. Global Constraints

A critical finding is that imposing constraints at every layer (layer-wise) significantly improves robustness compared to only constraining the final output. As shown in Figure 1(a), layer-wise constraints reduce the internal drift—the distance between high-precision and low-precision activations—by an order of magnitude.

Analysis of layer-wise constraints and dual variables. (a) Layer-wise MSE comparison between constraint types. (b) Correlation between train accuracy and the sum of dual variables. (c) Mixed-precision performance: selecting layers with the largest dual variables for full precision yields significantly higher accuracy.

Effective Mixed-Precision

The dual variables provide a principled way to perform mixed-precision quantization without expensive searches. Figure 1(c) demonstrates that by keeping only the layers with the highest dual variables in full precision, the model recovers nearly all the accuracy of a full-precision network while quantizing the majority of the layers. This confirms that $\lambda$ effectively identifies the "bottleneck" layers for quantization.

Performance Benchmarks

PDQAT consistently outperformed the popular DoReFa-net baseline. For instance, at 1-bit precision, PDQAT achieved an accuracy of 85.52%, surpassing the baseline's 84.91%.

Conclusion and Future Work

PDQAT offers a theoretically grounded alternative to heuristic-based QAT. By leveraging strong duality, it provides both superior accuracy and a built-in mechanism for sensitivity analysis via dual variables.

Future research directions include:

  • Granularity: Extending constraints from layer-wise to channel-wise quantization.
  • Scalability: Evaluating the approach on larger datasets like ImageNet and complex tasks like object detection.
  • Dynamic Allocation: Utilizing dual variables to automatically adjust bit-widths dynamically during the training process.

Summary updated

BibTeX
@inproceedings{hounie2023neural,
  author = {Ignacio Hounie and Juan Elenter and Alejandro Ribeiro},
  title = {{Neural Networks with Quantization Constraints}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094663},
}
Figure