ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks

Haoran You, Baopu Li, Huihong Shi, Yonggan Fu, Yingyan Lin

Abstract

Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are capable yet power hungry, impeding their more extensive deployment into resource-constrained devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks usually under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the efficacy of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score compared to state-of-the-art NN, while leading to up to 93% or 69% energy and latency savings, respectively.

Introduction

Modern neural networks, particularly Transformers and Convolutional Neural Networks (CNNs), are inherently "multiplication-hungry." While these operations enable high accuracy across diverse tasks, they are computationally expensive and power-intensive. This energy demand creates a significant barrier for deployment on resource-constrained edge devices, such as IoT sensors and smartphones.

To address this, researchers have explored "multiplication-free" networks, such as AdderNets (using additions) and ShiftNets (using bitwise shifts). While these alternatives are hardware-friendly, they typically suffer from a significant accuracy gap compared to standard models. Manually designing a "hybrid" network—one that strategically mixes standard multiplications with efficient operators—is a complex task. ShiftAddNAS automates this process, providing a hardware-inspired Neural Architecture Search (NAS) framework to design hybrid networks that optimize for both performance and efficiency.

Methodology

ShiftAddNAS introduces two primary innovations to bridge the gap between efficiency and accuracy in automated architecture design.

1. Hybrid Search Space

Traditional NAS frameworks typically search for architectural hyperparameters like kernel size or layer depth within a single operator type. ShiftAddNAS expands this by introducing a hybrid search space that includes four distinct building blocks:

  • Multi-head Self-Attention (Attn): For capturing global dependencies.
  • Convolution (Conv): For local feature extraction.
  • Bitwise Shift (Shift): For efficient scaling and movement.
  • Addition (Add): For low-energy feature aggregation.

This variety allows the search engine to select the most cost-effective operator for every specific layer in the network.

2. Heterogeneous Weight Sharing (HWS)

To accelerate the search process, NAS often utilizes a "supernet" where all potential sub-networks share a common set of weights. However, hybrid spaces present a unique challenge: different operators expect weights to follow different statistical distributions. For instance, Convolution weights typically follow a Gaussian distribution, whereas Adder weights follow a Laplacian distribution. Naive weight sharing forces a single distribution on all operators, leading to an "optimization gap" and poor performance.

ShiftAddNAS solves this through Heterogeneous Weight Sharing (HWS). It utilizes a shared weight pool but introduces a learnable transformation kernel. This kernel mathematically maps shared Gaussian weights into a Laplacian distribution specifically for the "Add" operators. This ensures each operator type functions optimally without significantly increasing the memory overhead of the search process.

Experimental Results

The framework was evaluated across both Computer Vision (CV) and Natural Language Processing (NLP) benchmarks, including ImageNet, CIFAR-10/100, and WMT’14 translation tasks.

Performance Benchmarks

  • ImageNet: The searched ShiftAddNAS-T1 model achieved 83.0% top-1 accuracy, outperforming the competitive BossNAS by +0.8% while consuming 30% less energy. Compared to pure multiplication-free networks, ShiftAddNAS improved accuracy by up to +7.7%.
  • NLP (WMT’14): ShiftAddNAS achieved a +4.9 higher BLEU score compared to the 8-bit Lite Transformer while reducing FLOPs by over 80%.

Hardware Efficiency

When mapped to specialized hardware (45nm CMOS and Xilinx FPGA), the searched models demonstrated transformative efficiency gains:

  • Up to 93% energy savings compared to standard full-precision Transformers.
  • Up to 69% latency savings on hardware accelerators.

Architectural Insights

The search process revealed distinct patterns in how hybrid networks should be structured. The optimal models tended to utilize standard Convolutions in early layers to capture raw features and Attention in later layers for global context. Shift and Add operators were strategically placed in the middle layers to drastically reduce the total energy footprint without sacrificing the model's representational power.

Conclusion and Future Work

ShiftAddNAS demonstrates that the trade-off between accuracy and energy efficiency is not a zero-sum game. By leveraging a hybrid search space and addressing the distribution mismatch in weight sharing, the framework produces models that are both more accurate and significantly more efficient than previous state-of-the-art designs.

While the current energy and latency savings are calculated based on specific hardware models (ASIC/FPGA), future work may explore optimizations for general-purpose CPUs or GPUs. Additionally, the authors suggest extending this hybrid approach to tasks like object detection and investigating the behavior of these networks during on-device fine-tuning.

Summary updated

Abstract

Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are capable yet power hungry, impeding their more extensive deployment into resource-constrained devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks usually under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the efficacy of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score compared to state-of-the-art NN, while leading to up to 93% or 69% energy and latency savings, respectively. Codes and pretrained models are available at https://github.com/RICE-EIC/ShiftAddNAS.
BibTeX
@inproceedings{you2022shiftaddnas,
  author = {Haoran You and Baopu Li and Huihong Shi and Yonggan Fu and Yingyan Lin},
  title = {{ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=CZZ7KWOP0-M},
}
Figure