Abstract
Emergent hardwares can support mixed precision CNN models inference that assign different bitwidths for different layers. Learning to find an optimal mixed precision model that can preserve accuracy and satisfy the specific constraints on model size and computation is extremely challenging due to the difficulty in training a mixed precision model and the huge space of all possible bit quantizations. In this paper, we propose a novel soft Barrier Penalty based NAS (BP-NAS) for mixed precision quantization, which ensures all the searched models are inside the valid domain defined by the complexity constraint, thus could return an optimal model under the given constraint by conducting search only one time. The proposed soft Barrier Penalty is differentiable and can impose very large losses to those models outside the valid domain while almost no punishment for models inside the valid domain, thus constraining the search only in the feasible domain. In addition, a differentiable Prob-1 regularizer is proposed to ensure learning with NAS is reasonable. A distribution reshaping training strategy is also used to make training more stable. BP-NAS sets new state of the arts on both classification (Cifar-10, ImageNet) and detection (COCO), surpassing all the efficient mixed precision methods designed manually and automatically. Particularly, BP-NAS achieves higher mAP (up to 2.7% mAP improvement) together with lower bit computation cost compared with the existing best mixed precision model on COCO detection.
Introduction
Quantizing neural networks—reducing the precision of weights and activations from 32-bit floats to lower bitwidths like 4-bit or 2-bit—is essential for deploying models on edge devices. However, not all layers are created equal: some are highly sensitive to precision loss, while others are redundant. Mixed-precision quantization exploits this by assigning more bits to sensitive layers and fewer to others.
The challenge is twofold. First, the search space is massive; for a network with $N$ layers and $M$ bitwidth options, there are $M^N$ possible configurations. Second, real-world applications usually have a "hard" budget (e.g., a specific model size or bit operation limit). Existing Neural Architecture Search (NAS) methods often treat this budget as a soft penalty term in the loss function, which frequently results in models that either miss the budget or are unnecessarily small, forcing researchers to restart the search multiple times with different penalty weights.
Methodology
BP-NAS (Barrier Penalty NAS) builds on the weight-sharing DARTS (Differentiable Architecture Search) approach to automate the search for mixed-precision quantization.
Search Space and Supernet
The framework utilizes a supernet where each layer contains multiple parallel operations representing different bitwidths. During the search, the algorithm learns the probability $p_{ij}$ of selecting a specific bitwidth for each layer.
Soft Barrier Penalty
The core innovation is the soft barrier penalty. Instead of a simple linear penalty for exceeding a budget, BP-NAS uses a logarithmic barrier function inspired by interior-point optimization. This function imposes almost no penalty when the model is well within the budget but creates an "infinite" loss as the model approaches the constraint boundary. This forces the search to stay within the feasible domain in a single search pass.
Prob-1 Regularizer and Distribution Reshaping
To ensure the search converges to a single bitwidth per layer, a differentiable Prob-1 regularizer encourages the probability distribution to become "one-hot." Additionally, because training low-bit models is unstable, the authors use a distribution reshaping strategy. This clips weight distributions to be more uniform before quantization, making the training process robust and preventing search failure.
Experiments and Results
The researchers evaluated BP-NAS across three standard benchmarks: CIFAR-10, ImageNet (ResNet-50), and COCO (Faster R-CNN).
On ImageNet, BP-NAS found a mixed-precision ResNet-50 model achieving 76.67% Top-1 accuracy within a 4-bit equivalent budget, outperforming previous methods like HAWQ and HAQ. The method was particularly effective for object detection on COCO, improving mAP by up to 2.7% compared to existing mixed-precision models while utilizing fewer bit operations (BOPs).
Unlike previous methods that required multiple search trials to hit a specific hardware target, BP-NAS consistently returned a valid, high-performing model in a single search execution, effectively allowing researchers to "search what they want" by setting a hard constraint ceiling.
Conclusion
BP-NAS provides a robust framework for mixed-precision quantization by transforming hard hardware constraints into differentiable barrier penalties. While the current BOPs metric is theoretical, future work could incorporate hardware-specific latency measurements. The method significantly reduces the manual effort required to balance model accuracy and hardware efficiency, setting new benchmarks for automated quantization.
Summary updated