Abstract
Emergent hardwares can support mixed precision CNN models inference that assign different bitwidths for different layers. Learning to find an optimal mixed precision model that can preserve accuracy and satisfy the specific constraints on model size and computation is extremely challenge due to the difficult in training a mixed precision model and the huge space of all possible bit quantizations. In this paper, we propose a novel soft Barrier Penalty based NAS (BP-NAS) for mixed precision quantization, which ensures all the searched models are inside the valid domain defined by the complexity constraint, thus could return an optimal model under the given constraint by conducting search only one time. The proposed soft Barrier Penalty is differentiable and can impose very large losses to those models outside the valid domain while almost no punishment for models inside the valid domain, thus constraining the search only in the feasible domain. In addition, a differentiable Prob-1 regularizer is proposed to ensure learning with NAS is reasonable. A distribution reshaping training strategy is also used to make training more stable. BP-NAS sets new state of the arts on both classification (Cifar-10, ImageNet) and detection (COCO), surpassing all the efficient mixed precision methods designed manually and automatically. Particularly, BP-NAS achieves higher mAP (up to 2.7% mAP improvement) together with lower bit computation cost compared with the existing best mixed precision model on COCO detection.
Introduction
Quantizing deep neural networks—reducing the bitwidth of weights and activations—is essential for deploying models on edge devices. While mixed-precision quantization (assigning different bitwidths to different layers) offers better accuracy-efficiency trade-offs than fixed-precision, it introduces two significant challenges. First, the search space is explosive; with $N$ layers and $M$ bitwidth options, there are $M^N$ possible configurations. Second, standard Neural Architecture Search (NAS) methods typically use a linear complexity penalty that often results in models that either exceed hardware budgets or are unnecessarily small, requiring tedious manual tuning of penalty weights.
BP-NAS (Barrier Penalty NAS) is a novel framework designed to find the optimal mixed-precision quantization for CNNs in a single pass. By treating hardware constraints as a hard boundary rather than a trade-off parameter, BP-NAS ensures the final model is both high-performing and strictly within the feasible domain of the hardware budget.
Methodology
Mixed-Precision Search Space
The authors build upon the weight-sharing DARTS (Differentiable Architecture Search) framework. In this setup, each layer is represented as a supernet cell where multiple quantization operations (e.g., 2-bit, 4-bit, 8-bit) are computed in parallel.
As shown in Figure 1, the architecture parameters $p_{ij}$ determine the importance of each bitwidth. The final output is a weighted sum, which allows the entire search process to be differentiable.
Soft Barrier Penalty
Inspired by interior-point optimization methods, the authors replace the standard linear complexity penalty with a differentiable "soft barrier penalty." Unlike linear penalties that provide a constant gradient, the barrier function remains near zero when the model is within the Bit Operations (BOPs) budget but approaches infinity as it nears the limit.
As illustrated in Figure 2, this creates a "soft wall" that keeps the search process inside the valid hardware constraint. By adjusting the $\mu$ parameter, the steepness of the barrier can be controlled, ensuring that the search algorithm "searches what you want" directly without exploring configurations that are far too large for the target hardware.
Prob-1 Regularizer and Distribution Reshaping
To ensure the final bitwidth selection is unambiguous, the authors introduce a Prob-1 regularizer. This differentiable term forces the architecture factors $p_{ij}$ to converge toward 0 or 1. Additionally, to combat the instability of training low-bit models, a Distribution Reshaping strategy is used during pre-training. By clipping weights and activations to follow a more uniform-like distribution, the model becomes more robust when quantized to very low bitwidths (like 2-bit).
Experimental Results
BP-NAS was evaluated across three standard benchmarks, consistently outperforming state-of-the-art manual and automated quantization methods:
- CIFAR-10: On ResNet20, BP-NAS achieved 92.30% accuracy with an 81.53× compression ratio, surpassing established methods like HAWQ and PACT.
- ImageNet: For ResNet50, it achieved 76.67% Top-1 accuracy under a 4-bit average constraint, outperforming HAQ and HAWQ by significant margins.
- COCO Detection: BP-NAS achieved a 0.358 mAP, a 2.7% improvement over previous best mixed-precision models, while maintaining a lower computational cost.
Conclusion
BP-NAS effectively addresses the instability and constraint-satisfaction issues in mixed-precision NAS. By utilizing a soft barrier penalty, the method ensures that hardware constraints are respected throughout the search process. While the memory cost of the supernet remains a consideration, the efficiency and accuracy gains—particularly in complex tasks like object detection—mark a significant step forward for automated model compression.
Summary updated