Simple Augmentation Goes a Long Way: ADRL for DNN Quantization

Lin Ning, Guoyang Chen, Weifeng Zhang, Xipeng Shen

Deep Neural Networks (DNNs) are computationally expensive, making them difficult to deploy on mobile devices and edge hardware. Quantization—the process of reducing the number of bits used to represent network weights and activations—is a primary technique to accelerate inference and reduce memory footprints. While "fixed-precision" quantization applies the same bit-width across an entire model, mixed-precision quantization is far more efficient, assigning more bits to highly sensitive layers and fewer bits to robust ones.

However, finding the optimal mixed-precision configuration is a "needle in a haystack" problem. For a network with dozens of layers, the search space of possible bit-width assignments grows exponentially. Deep Reinforcement Learning (DRL) has emerged as a popular tool to automate this search, but it suffers from severe instability. In the early stages of training, a standard DRL agent is essentially guessing, leading to high variance, slow convergence, and often suboptimal quantization strategies.

To solve this, researchers have introduced Augmented Deep Reinforcement Learning (ADRL). By adding a simple refinement step to the DRL agent's decision-making process, ADRL achieves a massive 4.5x to 64x speedup in finding optimal quantization strategies while maintaining or even improving model accuracy.

Methodology: The Augmented Policy Approximator

The core innovation of ADRL is a modification to the standard Actor-Critic architecture (specifically DDPG) used in prior quantization work like HAQ (Hardware-Aware Quantization). The authors replace the standard actor with an Augmented Policy Approximator.

Architecture of the Augmented Policy Approximator

This augmented approximator consists of two main components:

  1. Expanded Actor: Instead of outputting a single "best" action (bit-width) for a given layer, the neural network is designed to output $k$ candidate actions.
  2. Refinement Function (Q-value Indicator): A non-learnable "guide" evaluates these $k$ candidates and selects the most promising one to actually execute in the environment.

The researchers tested two types of Q-value Indicators to guide the agent:

  • Profiling-based: Runs a quick inference on a small subset of data to determine which bit-width preserves the most accuracy for that specific layer.
  • Distance-based: Calculates the mathematical "distance" (such as L2 norm or KL-divergence) between the original weights and the quantized weights. A smaller distance indicates a better candidate.

To further accelerate the process, ADRL employs memoization (caching previous evaluation results) and an early termination strategy that halts the search once the agent's performance stabilizes.

Experimental Setup and Key Insights

The researchers evaluated ADRL on two major image classification benchmarks: CIFAR-10 (using CifarNet and ResNet20) and ImageNet (using AlexNet and ResNet50). This provided a comprehensive test across both small-scale problems and large-scale, industry-standard models.

A key theoretical insight from the paper is that ADRL effectively stabilizes the "cold start" problem inherent to DRL. Because the Q-value indicator acts as a "safety rail," the agent takes reasonably good actions even before its policy network has fully learned the environment. The authors prove analytically that this augmentation reduces the variance of estimated Q-values and increases the effective step size of updates to the Critic network, allowing the agent to learn from higher-quality data much faster.

Key Results

ADRL significantly outperformed the state-of-the-art DRL-based method, HAQ, across multiple metrics:

  • Search Speed: ADRL reached optimal configurations 4.5x to 64x faster than HAQ. For example, on ResNet20, the search time dropped from over 6 hours (22,680 seconds) to just under 6 minutes (342 seconds).
  • Accuracy: On the large-scale ResNet50 model (ImageNet), ADRL achieved a 10x compression ratio with 0% accuracy loss. In contrast, the baseline HAQ suffered a 1.16% drop in accuracy for a similar compression level.
  • Stability: The "Coefficient of Variance" (a measure of instability during the search) was consistently lower for ADRL across all models, proving the search process is much more reliable.
Comparison of search and finetuning times between HAQ and P-ADRL

Limitations and Future Work

While highly effective, the current implementation of ADRL focuses strictly on layer-wise quantization. Extending this approach to finer-grained kernel-wise quantization remains an area for future exploration.

Additionally, there is a trade-off in indicator selection: the profiling-based indicator is highly effective but requires a small validation set, whereas the "data-free" distance-based indicator is slightly less accurate. Finally, while the authors suggest ADRL's core concept could be applied to other RL domains like robotics or game-playing, designing the specific "augmentation schemes" (the indicators) for those fields would likely require significant domain expertise.

Summary updated

Abstract

Mixed precision quantization improves DNN performance by assigning different layers with different bit-width values. Searching for the optimal bit-width for each layer, however, remains a challenge. Deep Reinforcement Learning (DRL) shows some recent promise. It however suffers instability due to function approximation errors, causing large variances in the early training stages, slow convergence, and suboptimal policies in the mixed-precision quantization problem. This paper proposes augmented DRL (ADRL) as a way to alleviate these issues. This new strategy augments the neural networks in DRL with a complementary scheme to boost the performance of learning. The paper examines the effectiveness of ADRL both analytically and empirically, showing that it can produce more accurate quantized models than the state of the art DRL-based quantization while improving the learning speed by 4.5-64 times.
BibTeX
@inproceedings{ning2021simple,
  author = {Lin Ning and Guoyang Chen and Weifeng Zhang and Xipeng Shen},
  title = {{Simple Augmentation Goes a Long Way: ADRL for DNN Quantization}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure