DNN Quantization with Attention

Ghouthi Boukli Hacene, Lukas Mauch, Stefan Uhlich, Fabien CardinauxICLR 2021 · OpenReview

TL;DR

This paper introduces DNN Quantization with Attention (DQA), a novel training method that improves the accuracy of low-bit quantized neural networks. DQA uses an attention mechanism to dynamically blend multiple quantization precisions during training, allowing the network to learn the optimal quantization strategy and significantly reduce accuracy loss compared to existing methods.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have achieved state-of-the-art results in many computer vision tasks. However, their large memory footprint and computational demands make them unsuitable for resource-constrained embedded systems and mobile applications. Quantization, which reduces the number of bits used to represent network weights and activations, is a key technique for addressing this. While effective, low-bit quantization (e.g., 1-bit or 2-bit) often leads to a significant drop in accuracy, especially for complex tasks or already compact network architectures. Existing methods like Binary-Relax (BR) attempt to mitigate this by interpolating between quantized and full-precision values, but their handcrafted scaling factors can limit convergence. This paper aims to bridge the gap between aggressive quantization for efficiency and maintaining high accuracy.

Methods

The core innovation is DNN Quantization with Attention (DQA). DQA introduces a training procedure that doesn't rely on a single quantization method but instead uses a learnable linear combination of multiple quantization functions. Specifically:

  • Multiple Quantizers: DQA considers a set of $K$ different quantization functions, $Q_k(w; n_k)$, each with a different bitwidth $n_k$. These are sorted by bitwidth ($n_1 < n_2 < \dots < n_K$).
  • Attention Mechanism: An attention vector $\alpha \in [0, 1]^K$ is introduced, where each element $\alpha_k$ represents the importance of the corresponding quantization function $Q_k$. This attention vector is learned during training.
  • Softmax with Temperature: The attention values are generated using a softmax function with a temperature parameter $T$. This allows for a smooth transition.
  • Temperature Scheduling: The temperature $T$ is exponentially cooled down during training. Initially, it's high, leading to uniform attention values (all quantizations are considered equally). As training progresses, the temperature decreases, causing the attention to focus on a single, typically low-bit, quantization function.
  • Loss Regularization: To explicitly encourage low-bit quantization, a penalty term $\lambda$ is added to the loss function, penalizing larger bitwidths.
  • Flexibility: DQA is agnostic to the specific quantization functions used, allowing it to be combined with existing methods like min-max, Statistics-Aware Weight Binning (SAWB), Binary Weight Network (BWN), and Ternary Weight Network (TWN).

Datasets

The paper evaluates DQA on the following standard object recognition datasets:

  • CIFAR10
  • CIFAR100
  • ImageNet ILSVRC 2012

Key Insights and Ideas

  • Learning the Quantization Strategy: The most significant insight is that allowing the network to learn which quantization precision to use at different stages of training, rather than fixing it, leads to better accuracy. The attention mechanism and temperature scheduling enable this dynamic adaptation.
  • Smoother Loss Landscape: The gradual convergence from multiple quantizations to a single one, facilitated by the temperature schedule, effectively smooths the loss function. This makes it easier for the network to converge to a good quantized state, reducing the accuracy drop.
  • Synergy with Existing Methods: DQA doesn't replace existing quantization techniques but enhances them. By combining them in a learnable way, it leverages their strengths and mitigates their individual weaknesses.
  • Effective for Lightweight Architectures: DQA shows particular promise in reducing the accuracy drop for lightweight networks like MobileNetV2, which are inherently harder to compress without significant performance degradation.

Key Results

  • Improved Accuracy: DQA consistently outperforms other low-bit quantization techniques (including the baseline quantized network and Binary-Relax) on CIFAR10, CIFAR100, and ImageNet.
  • Near Full-Precision Accuracy: DQA-trained networks achieve accuracy very close to their full-precision counterparts, even with aggressive 1-bit or 2-bit quantization. For example, on CIFAR100 with ResNet18, DQA achieved 76.4% accuracy with 2-bit min-max quantization, compared to 70.0% for the baseline 2-bit min-max.
  • Reduced Accuracy Drop: DQA significantly reduces the accuracy drop when quantizing lightweight architectures like MobileNetV2. For instance, on ImageNet, DQA with 2-bit min-max quantization achieved 52.2% accuracy for MobileNetV2, a substantial improvement over the baseline 2-bit min-max at 44.2%.
  • Effective Attention Evolution: Experiments show that attention values indeed start uniformly and converge towards a single low-bit quantization function by the end of training, as visualized in Figure 2.

Limitations and Future Work

  • Computational Overhead: While DQA aims to reduce computational complexity in the final quantized model, the training process itself involves managing multiple quantizers and attention mechanisms, which might introduce a slight overhead during training compared to single-method quantization.
  • Hyperparameter Tuning: The temperature schedule parameters (initial temperature $T(0)$, decay rate $\Psi$) and the penalty term $\lambda$ are hyperparameters that might require tuning for optimal performance on different architectures and datasets.
  • Future Work: The authors suggest that DQA is a promising extension to existing quantization methods and could be applied to other network architectures and tasks. Further exploration into more complex attention mechanisms or different temperature scheduling strategies could also be beneficial.

Summary updated

Abstract

Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). However, low-bit quantization can also cause a considerable drop in accuracy, in particular when we apply it to complex learning tasks or lightweight DNN architectures. In this paper, we propose a training procedure that relaxes the low-bit quantization. We call this procedure \textit{DNN Quantization with Attention} (DQA). The relaxation is achieved by using a learnable linear combination of high, medium and low-bit quantizations. Our learning procedure converges step by step to a low-bit quantization using an attention mechanism with temperature scheduling. In experiments, our approach outperforms other low-bit quantization techniques on various object recognition benchmarks such as CIFAR10, CIFAR100 and ImageNet ILSVRC 2012, achieves almost the same accuracy as a full precision DNN, and considerably reduces the accuracy drop when quantizing lightweight DNN architectures.
BibTeX
@inproceedings{hacene2021dnn,
  author = {Ghouthi Boukli Hacene and Lukas Mauch and Stefan Uhlich and Fabien Cardinaux},
  title = {{DNN Quantization with Attention}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=Megq85LqLi},
}
Figure