ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks

Haoran You, Baopu Li, Huihong Shi, Yingyan Lin

Introduction

Neural networks with intensive multiplications, such as standard convolutions and transformers, are powerful but notoriously power-hungry. This computational cost impedes their deployment on resource-constrained edge devices. To address this, ShiftAddNAS introduces a novel Neural Architecture Search (NAS) framework designed to build hybrid neural networks. These networks intelligently mix standard multiplication-based operators (like convolutions and attention) with highly efficient multiplication-free operators (like bitwise shifts and additions). To make searching this diverse space computationally feasible, the authors propose a "heterogeneous weight sharing" strategy that aligns the fundamentally different weight distributions of these operators, resulting in models that achieve state-of-the-art accuracy-efficiency trade-offs on both vision and language tasks.

The Challenge of Hybrid Architectures

Standard neural networks rely heavily on floating-point multiplications, which dominate energy consumption and latency. To deploy models on edge devices, researchers have developed "multiplication-free" networks (e.g., AdderNets or ShiftNets) that replace multiplications with cheaper bitwise shifts and additions.

Energy cost comparison of multiplication versus shift and add operations across different hardware and precision formats.

However, purely multiplication-free networks usually suffer a noticeable drop in accuracy. The logical solution is a hybrid network: use powerful multiplications where they matter most, and cheap shifts/adds everywhere else. Neural Architecture Search (NAS) is the perfect tool to find this optimal mix.

Unfortunately, standard one-shot NAS relies on weight sharing—where sub-networks inherit weights from a giant "supernet" to save memory—which breaks down in a hybrid search space. Different operators naturally learn fundamentally different weight distributions. For example, Convolution weights typically follow a Gaussian distribution, while Add operator weights follow a Laplacian distribution.

Weight distributions for Convolution (Gaussian) and Add (Laplacian) operators, illustrating the mismatch that complicates standard weight sharing.

Forcing these heterogeneous operators to share the exact same raw weights ruins the training process, leading to a supernet that poorly estimates the true performance of the sub-networks.

The ShiftAddNAS Solution

To solve the weight-sharing dilemma, ShiftAddNAS introduces two main components:

1. A Hardware-Inspired Hybrid Search Space: The search space includes four types of blocks: Attention (Attn), Convolution (Conv), Shift, and Add. The supernet is designed with multi-resolution pipelines for Computer Vision (CV) and multi-branch structures for Natural Language Processing (NLP).

2. Heterogeneous Weight Sharing (HWS): This is the core algorithmic novelty. Instead of forcing all operators to use the exact same weights, the supernet maintains a shared weight pool and adapts it for each operator:

  • Conv blocks use the weights directly (encouraged to be Gaussian via a KL-divergence loss).
  • Shift blocks quantize the shared weights to powers of two.
  • Add blocks use a learnable transformation kernel—a lightweight, piece-wise linear transformation that maps the shared Gaussian weights into the Laplacian distribution that Add operators prefer.

After training the supernet with HWS, the authors use an evolutionary search algorithm to find the best sub-networks under specific hardware constraints (like FLOPs or latency). By using a simple piece-wise linear transformation kernel, the supernet can maintain a single set of weights while serving each operator its preferred distribution. This solves the one-shot NAS ranking dilemma and shrinks the supernet size by over 40% compared to a non-sharing approach, without sacrificing search accuracy.

Key Results and Performance

ShiftAddNAS was evaluated across both Natural Language Processing (NLP) and Computer Vision (CV) tasks, demonstrating significant improvements over existing baselines.

  • NLP Tasks (WMT'14 En-Fr & En-De): ShiftAddNAS consistently outperforms baselines. When quantized to 8-bit, it achieves up to a +4.9 higher BLEU score compared to the Lite Transformer (8-bit). Compared to a standard full-precision Transformer, it saves ~69% in energy and latency while maintaining a comparable BLEU score.
  • CV Tasks (ImageNet, CIFAR): On ImageNet, ShiftAddNAS achieves up to +7.7% higher accuracy compared to state-of-the-art purely multiplication-free CNNs. It also beats highly competitive hybrid NAS baselines like BossNAS, offering a +0.8% accuracy boost alongside 30% energy savings.
  • Search Efficiency: Thanks to HWS, the search cost is kept highly competitive, requiring about 9.3 GPU days for NLP tasks and 8.9 GPU days for CV tasks on a single NVIDIA V100 GPU.

Conclusion and Future Work

Hybrid architectures represent a sweet spot in neural network design: mixing multiplications with shifts and adds yields a significantly better Pareto frontier than using either paradigm exclusively. The search algorithm naturally tends to place Conv blocks early in the network and Attn blocks later, filling the gaps with efficient Shift and Add blocks.

While the ~9 GPU days search cost remains a non-trivial computational barrier for smaller labs, and the hardware metrics are tied to specific profiles (e.g., 45nm CMOS/FPGA), the framework opens up a highly promising perspective for searching accurate and efficient NNs. The heterogeneous weight-sharing technique could easily be expanded to accommodate an even wider variety of exotic, hardware-efficient operators in the future.

Summary updated

Abstract

Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are powerful yet power hungry, impeding their more extensive deployment into resource-constrained edge devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks in general under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the effectiveness of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score as compared to state-of-the-art expert-designed and neural architecture searched NNs, while leading to up to 93% or 69% energy and latency savings, respectively. All the codes will be released upon acceptance.
BibTeX
@inproceedings{you2022shiftaddnas,
  author = {Haoran You and Baopu Li and Huihong Shi and Yingyan Lin},
  title = {{ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=CZZ7KWOP0-M},
}
Figure