Deep Neural Networks (DNNs) are computationally expensive, making them difficult to deploy on mobile devices and edge hardware. Quantization—the process of reducing the number of bits used to represent network weights and activations—is a primary technique to accelerate inference and reduce memory footprints. While "fixed-precision" quantization applies the same bit-width across an entire model, mixed-precision quantization is far more efficient, assigning more bits to highly sensitive layers and fewer bits to robust ones.
However, finding the optimal mixed-precision configuration is a "needle in a haystack" problem. For a network with dozens of layers, the search space of possible bit-width assignments grows exponentially. Deep Reinforcement Learning (DRL) has emerged as a popular tool to automate this search, but it suffers from severe instability. In the early stages of training, a standard DRL agent is essentially guessing, leading to high variance, slow convergence, and often suboptimal quantization strategies.
To solve this, researchers have introduced Augmented Deep Reinforcement Learning (ADRL). By adding a simple refinement step to the DRL agent's decision-making process, ADRL achieves a massive 4.5x to 64x speedup in finding optimal quantization strategies while maintaining or even improving model accuracy.
Methodology: The Augmented Policy Approximator
The core innovation of ADRL is a modification to the standard Actor-Critic architecture (specifically DDPG) used in prior quantization work like HAQ (Hardware-Aware Quantization). The authors replace the standard actor with an Augmented Policy Approximator.
This augmented approximator consists of two main components:
- Expanded Actor: Instead of outputting a single "best" action (bit-width) for a given layer, the neural network is designed to output $k$ candidate actions.
- Refinement Function (Q-value Indicator): A non-learnable "guide" evaluates these $k$ candidates and selects the most promising one to actually execute in the environment.
The researchers tested two types of Q-value Indicators to guide the agent:
- Profiling-based: Runs a quick inference on a small subset of data to determine which bit-width preserves the most accuracy for that specific layer.
- Distance-based: Calculates the mathematical "distance" (such as L2 norm or KL-divergence) between the original weights and the quantized weights. A smaller distance indicates a better candidate.
To further accelerate the process, ADRL employs memoization (caching previous evaluation results) and an early termination strategy that halts the search once the agent's performance stabilizes.
Experimental Setup and Key Insights
The researchers evaluated ADRL on two major image classification benchmarks: CIFAR-10 (using CifarNet and ResNet20) and ImageNet (using AlexNet and ResNet50). This provided a comprehensive test across both small-scale problems and large-scale, industry-standard models.
A key theoretical insight from the paper is that ADRL effectively stabilizes the "cold start" problem inherent to DRL. Because the Q-value indicator acts as a "safety rail," the agent takes reasonably good actions even before its policy network has fully learned the environment. The authors prove analytically that this augmentation reduces the variance of estimated Q-values and increases the effective step size of updates to the Critic network, allowing the agent to learn from higher-quality data much faster.
Key Results
ADRL significantly outperformed the state-of-the-art DRL-based method, HAQ, across multiple metrics:
- Search Speed: ADRL reached optimal configurations 4.5x to 64x faster than HAQ. For example, on ResNet20, the search time dropped from over 6 hours (22,680 seconds) to just under 6 minutes (342 seconds).
- Accuracy: On the large-scale ResNet50 model (ImageNet), ADRL achieved a 10x compression ratio with 0% accuracy loss. In contrast, the baseline HAQ suffered a 1.16% drop in accuracy for a similar compression level.
- Stability: The "Coefficient of Variance" (a measure of instability during the search) was consistently lower for ADRL across all models, proving the search process is much more reliable.
Limitations and Future Work
While highly effective, the current implementation of ADRL focuses strictly on layer-wise quantization. Extending this approach to finer-grained kernel-wise quantization remains an area for future exploration.
Additionally, there is a trade-off in indicator selection: the profiling-based indicator is highly effective but requires a small validation set, whereas the "data-free" distance-based indicator is slightly less accurate. Finally, while the authors suggest ADRL's core concept could be applied to other RL domains like robotics or game-playing, designing the specific "augmentation schemes" (the indicators) for those fields would likely require significant domain expertise.
Summary updated