Introduction
Neural networks with intensive multiplications, such as standard convolutions and transformers, are powerful but notoriously power-hungry. This computational cost impedes their deployment on resource-constrained edge devices. To address this, ShiftAddNAS introduces a novel Neural Architecture Search (NAS) framework designed to build hybrid neural networks. These networks intelligently mix standard multiplication-based operators (like convolutions and attention) with highly efficient multiplication-free operators (like bitwise shifts and additions). To make searching this diverse space computationally feasible, the authors propose a "heterogeneous weight sharing" strategy that aligns the fundamentally different weight distributions of these operators, resulting in models that achieve state-of-the-art accuracy-efficiency trade-offs on both vision and language tasks.
The Challenge of Hybrid Architectures
Standard neural networks rely heavily on floating-point multiplications, which dominate energy consumption and latency. To deploy models on edge devices, researchers have developed "multiplication-free" networks (e.g., AdderNets or ShiftNets) that replace multiplications with cheaper bitwise shifts and additions.
However, purely multiplication-free networks usually suffer a noticeable drop in accuracy. The logical solution is a hybrid network: use powerful multiplications where they matter most, and cheap shifts/adds everywhere else. Neural Architecture Search (NAS) is the perfect tool to find this optimal mix.
Unfortunately, standard one-shot NAS relies on weight sharing—where sub-networks inherit weights from a giant "supernet" to save memory—which breaks down in a hybrid search space. Different operators naturally learn fundamentally different weight distributions. For example, Convolution weights typically follow a Gaussian distribution, while Add operator weights follow a Laplacian distribution.
Forcing these heterogeneous operators to share the exact same raw weights ruins the training process, leading to a supernet that poorly estimates the true performance of the sub-networks.
The ShiftAddNAS Solution
To solve the weight-sharing dilemma, ShiftAddNAS introduces two main components:
1. A Hardware-Inspired Hybrid Search Space:
The search space includes four types of blocks: Attention (Attn), Convolution (Conv), Shift, and Add. The supernet is designed with multi-resolution pipelines for Computer Vision (CV) and multi-branch structures for Natural Language Processing (NLP).
2. Heterogeneous Weight Sharing (HWS): This is the core algorithmic novelty. Instead of forcing all operators to use the exact same weights, the supernet maintains a shared weight pool and adapts it for each operator:
- Conv blocks use the weights directly (encouraged to be Gaussian via a KL-divergence loss).
- Shift blocks quantize the shared weights to powers of two.
- Add blocks use a learnable transformation kernel—a lightweight, piece-wise linear transformation that maps the shared Gaussian weights into the Laplacian distribution that Add operators prefer.
After training the supernet with HWS, the authors use an evolutionary search algorithm to find the best sub-networks under specific hardware constraints (like FLOPs or latency). By using a simple piece-wise linear transformation kernel, the supernet can maintain a single set of weights while serving each operator its preferred distribution. This solves the one-shot NAS ranking dilemma and shrinks the supernet size by over 40% compared to a non-sharing approach, without sacrificing search accuracy.
Key Results and Performance
ShiftAddNAS was evaluated across both Natural Language Processing (NLP) and Computer Vision (CV) tasks, demonstrating significant improvements over existing baselines.
- NLP Tasks (WMT'14 En-Fr & En-De): ShiftAddNAS consistently outperforms baselines. When quantized to 8-bit, it achieves up to a +4.9 higher BLEU score compared to the Lite Transformer (8-bit). Compared to a standard full-precision Transformer, it saves ~69% in energy and latency while maintaining a comparable BLEU score.
- CV Tasks (ImageNet, CIFAR): On ImageNet, ShiftAddNAS achieves up to +7.7% higher accuracy compared to state-of-the-art purely multiplication-free CNNs. It also beats highly competitive hybrid NAS baselines like BossNAS, offering a +0.8% accuracy boost alongside 30% energy savings.
- Search Efficiency: Thanks to HWS, the search cost is kept highly competitive, requiring about 9.3 GPU days for NLP tasks and 8.9 GPU days for CV tasks on a single NVIDIA V100 GPU.
Conclusion and Future Work
Hybrid architectures represent a sweet spot in neural network design: mixing multiplications with shifts and adds yields a significantly better Pareto frontier than using either paradigm exclusively. The search algorithm naturally tends to place Conv blocks early in the network and Attn blocks later, filling the gaps with efficient Shift and Add blocks.
While the ~9 GPU days search cost remains a non-trivial computational barrier for smaller labs, and the hardware metrics are tied to specific profiles (e.g., 45nm CMOS/FPGA), the framework opens up a highly promising perspective for searching accurate and efficient NNs. The heterogeneous weight-sharing technique could easily be expanded to accommodate an even wider variety of exotic, hardware-efficient operators in the future.
Summary updated