Hyperdrive: A Multi-Chip Systolically Scalable Binary-Weight CNN Inference Engine

Renzo Andri, Lukas Cavigelli, Davide Rossi, Luca Benini

Abstract

Hyperdrive is a novel hardware accelerator designed to bring high-resolution computer vision to power-constrained IoT devices. By implementing a feature-map stationary dataflow—streaming 1-bit binary weights while keeping feature maps on-chip—and a systolically scalable multi-chip architecture, Hyperdrive achieves a system-level energy efficiency of 4.3 TOp/s/W. This represents a 3.1x improvement over state-of-the-art accelerators, enabling the deployment of complex models like ResNet and YOLOv3 in milliwatt-scale environments.

Introduction

Deep Neural Networks (DNNs) are essential for modern computer vision but pose significant challenges for edge deployment due to high power consumption. Binary-Weight Networks (BWNs) mitigate memory constraints by compressing weights to 1 bit. However, traditional accelerators often hit an "energy wall" caused by excessive I/O energy spent moving large feature maps (FMs) between on-chip and off-chip memory. Most existing designs either stream FMs, consuming massive I/O energy, or store everything on-chip, which limits the device to low-resolution models. Hyperdrive addresses this by flipping the dataflow priority and introducing a scalable multi-chip design.

Architecture and Methodology

The core innovation of Hyperdrive is its Feature-Map Stationary approach. In a typical BWN, activations (feature maps) are significantly larger than the 1-bit weights. Hyperdrive keeps these large feature maps in on-chip memory and "streams" the compressed binary weights from external memory. This "I/O flip" reduces system-level energy consumption by approximately 25% compared to designs that move activations.

Figure 1: High-level architecture of the Hyperdrive chip, featuring a grid of Tile Processing Units (TPUs) and a dedicated border interface for multi-chip scalability.

The system is systolically scalable, allowing multiple chips to be arranged in a 2D mesh. Each chip processes a spatial "tile" of the input image and exchanges border pixels through a dedicated interface. This allows the system to scale to arbitrarily large images by adding more low-cost chips, making it suitable for applications ranging from simple doorbell cameras to high-end autonomous drones.

Figure 3: Illustration of mapping feature maps onto tiles and streaming weights from a central buffer, supporting 3D tiling for large-scale inference.

Each chip contains a 7x7 grid of Tile Processing Units (TPUs). These units utilize FP16 (16-bit floating point) for activations. While more resource-intensive than fixed-point math, FP16 provides the robustness needed for deep networks, and the FM-stationary approach makes the extra bits affordable in terms of energy. A latency-hiding dataflow mechanism using ping-pong buffering ensures that weights are loaded while computation proceeds, maintaining high hardware utilization.

Experimental Evaluation

Hyperdrive was evaluated using industry-standard benchmarks, including ImageNet (for ResNet-18/34) and COCO (for YOLOv3 object detection). Implemented in 22nm FDX technology, a single chip reaches a throughput of 88 GOp/s at 0.5V.

Key results include:

  • Efficiency: Achieved 4.3 TOp/s/W at the system level, a 3.1x improvement over previous state-of-the-art BWN accelerators.
  • Utilization: Maintained over 97% utilization for ResNet-34, minimizing idle hardware cycles.
  • Scalability: Simulations demonstrated that a 20x10 mesh of chips could execute a ResNet-152 model on high-definition (2k x 1k) images while maintaining high efficiency.

Discussion and Conclusion

Hyperdrive proves that optimizing data movement is as critical as optimizing computation for edge AI. By treating weights as the transient stream and activations as stationary, it overcomes the I/O energy-induced efficiency wall. While the current design is optimized for 1x1 and 3x3 convolutions—requiring external handling for larger kernels like the initial 7x7 layer in ResNet—it offers a highly scalable and efficient solution for bringing advanced computer vision to the IoT frontier. Future work may address kernel size constraints and explore physical packaging impacts on I/O energy.

Summary updated

Abstract

Deep neural networks have achieved impressive results in computer vision and machine learning. Unfortunately, state-of-the-art networks are extremely compute and memory intensive, which makes them unsuitable for mW-devices such as IoT end-nodes. Aggressive quantization of these networks dramatically reduces the computation and memory footprint. Binary-weight neural networks (BWNs) follow this trend, pushing weight quantization to the limit. Hardware accelerators for BWNs presented up to now have focused on core efficiency, disregarding I/O bandwidth, and system-level efficiency that are crucial for the deployment of accelerators in ultra-low power devices. We present Hyperdrive: a BWN accelerator dramatically reducing the I/O bandwidth exploiting a novel binary-weight streaming approach, which can be used for an arbitrarily sized convolutional neural network architecture and input resolution by exploiting the natural scalability of the compute units both at chip-level and system-level by arranging Hyperdrive chips systolically in a 2D mesh while processing the entire feature map together in parallel. Hyperdrive achieves 4.3 TOp/s/W system-level efficiency (i.e., including I/Os)-3.1× higher than state-of-the-art BWN accelerators, even if its core uses resource-intensive FP16 arithmetic for increased robustness.
BibTeX
@article{andri2019hyperdrive,
  author = {Renzo Andri and Lukas Cavigelli and Davide Rossi and Luca Benini},
  title = {{Hyperdrive: A Multi-Chip Systolically Scalable Binary-Weight CNN Inference Engine}},
  journal = {IEEE J. Emerg. Sel. Topics Circuits Syst.},
  year = {2019},
  doi = {10.1109/jetcas.2019.2905654},
}
Figure