When executing deep neural networks on traditional hardware, processors typically operate like a single, highly optimized workshop. The hardware takes the first layer of a network, computes its outputs for a batch of data, writes those intermediate results back to off-chip memory (DRAM), reconfigures its internal logic for the second layer, reads the data back, and repeats the process. While flexible, this temporal reuse strategy hits a severe bottleneck known as the memory wall: moving millions of intermediate activations back and forth between the compute cores and external memory consumes the vast majority of the system's time and energy.
To overcome this, a prominent research direction focuses on Hardware Streaming Architectures (HSAs) driven by spatial pipelining. Instead of forcing different layers to take turns on the same piece of silicon, this approach maps the neural network graph directly onto physical space. Every layer—or segment of a layer—is assigned its own dedicated hardware stage, creating a continuous, on-chip assembly line.
Bypassing the Memory Wall
The primary reason spatial pipelining achieves orders-of-magnitude improvements in throughput and energy efficiency is that it fundamentally changes how data moves. In a layer-dedicated streaming architecture, the physical output of the hardware block for Layer $N$ is wired directly into the physical input of the block for Layer $N+1$.
As surveyed by Stylianos I. Venieris et al., this structural choice completely separates streaming architectures from single computation engines. By instantiating a dedicated hardware stage for every layer in the network, intermediate feature maps can reside entirely within the chip's internal routing and local buffers.
The implications of this are demonstrated clearly across various hardware implementations:
- Junsong Wang et al. show that keeping feature maps entirely on-chip eliminates the latency and energy costs associated with external memory access, effectively bypassing the memory wall.
- Linh Ngoc Le et al. leverage this coarse-grained pipelining to maximize throughput for continuous data streams, passing data directly between layers without intermediate off-chip storage.
- The framework proposed in LUTMUL breaks through conventional hardware roofline limits by ensuring intermediate activations remain strictly within the internal routing and buffers of the device.
Fine-Grained Dataflow and Inter-Layer Parallelism
If a spatial pipeline required Layer 2 to wait until Layer 1 finished processing an entire image, the hardware would suffer from massive buffer requirements and high idle times. Spatial pipelining works efficiently because it operates on a fine-grained, continuous dataflow.
Layer 2 does not wait for the whole image; it begins processing the moment Layer 1 produces a single output pixel, row, or data tile. This creates inter-layer parallelism, where every single layer of the neural network is actively processing a different data sample or a different spatial region of the same sample simultaneously.
Traditional (Temporal):
[ Input 1 -> Layer 1 ] -> Write to DRAM -> [ Read from DRAM -> Layer 2 ]
Spatial Pipeline (Streaming):
Input 3 -> [ Layer 1 Hardware ] -> (On-Chip Buffer) -> [ Layer 2 Hardware ] -> Output 1
^ ^
Processing Sample 3 Processing Sample 2
Martino Dazzi et al. explain that by transmitting partial results to the next stage as soon as they are generated, the system minimizes idle time and treats the hardware array as a physical manifestation of the neural network graph. To manage this continuous flow, architectures like the one designed by Chao Wang et al. organize the hardware into multi-stage pipelines of dedicated units (e.g., matrix multiplication, partial sum accumulation, and activation functions) to process data tiles concurrently.
Furthermore, as noted by Xiang Shui-Ying et al., this spatial distribution simplifies control logic. Because each piece of hardware has a fixed, dedicated role, the system avoids the complex scheduling and reconfiguration overhead required when a single execution engine tries to handle multiple distinct layers sequentially.
Managing Complex Topologies
While an assembly line is straightforward for a linear network, modern neural networks feature complex topologies, such as branching paths and skip connections (e.g., ResNets or YOLO models). If Layer 1 needs to send its output to both Layer 2 and a downstream Layer 4, a naive pipeline would stall because Layer 4 cannot accept the data until Layer 2 and Layer 3 finish their work.
Spatial architectures solve this synchronization problem using two primary mechanisms: delay buffers and configurable crossbars.
Delay Buffers
To handle skip connections without halting the pipeline, designers insert hardware delay buffers into the shortcut path. These buffers act as physical "holding pens" that delay the skipped data by the exact number of clock cycles it takes for the parallel convolutional path to complete. Chaim Baskin et al. demonstrated that matching the internal pipeline latency this way allows skipped data to be merged seamlessly with processed data without stalling the dataflow or requiring complex control logic. For larger models where these buffers might overwhelm on-chip memory, Petros Toupas showed that toolflows can intelligently offload oversized skip-connection buffers to external memory while keeping the primary model weights on-chip to maintain ultra-low latency.
Configurable Crossbars
To accommodate non-linear routing and diverse layer types, modern streaming architectures use flexible interconnects. Petros Toupas et al. utilized configurable crossbars to connect heterogeneous hardware nodes (such as convolution and pooling layers), enabling flexible data routing while ensuring that intermediate processing steps never require off-chip memory access.
Generalizing the Streaming Paradigm
Though highly popular in FPGA design for real-time edge applications—such as the hand pose estimation system implemented by Mhd Rashed Al Koutayni et al.—the concept of spatial pipelining is a broad architectural paradigm that spans multiple hardware substrates and domains:
- Modular and Sparse Dataflows: Single-author work by Yizhao Gao demonstrates that streaming pipelines can be made composable. By utilizing a library of parameterized components that communicate via a unified interface carrying both feature data and spatial coordinates, hardware can dynamically process sparse event-camera data without losing its inherent sparsity.
- Mixed-Signal VLSI: The core intuition of splitting calculations into pipelined stages even operates at the circuit level. Masakazu Yagi et al. implemented an asynchronous median-detection architecture that splits multi-bit calculations into upper and lower bit segments, allowing the hardware to process new image blocks while simultaneously finalizing calculations for previous segments.
- Multi-GPU Distributed Systems: The concept scales up from single chips to massive clusters. Early foundations by Stephen S. Wilson envisioned arranging multiple hardware boards in a pipeline to execute deep architectures sequentially. Modernizing this concept for distributed systems, Enrico Meloni et al. developed a method for continuous pipeline interleaving across multiple GPUs. By dividing a network into sequential stages across separate devices and interleaving forward passes, backward passes, and weight updates without flushing the pipeline, they eliminate the idle time typically associated with batch-based processing, keeping distributed hardware fully utilized for continuous data streams.
Key Foundational Work
Neural Computing on a One Dimensional SIMD Array (1989) — contributed 1 concept elements to this direction.
A Mixed-Signal VLSI for Real-Time Generation of Edge-Based Image Vectors (2003) — contributed 1 concept elements to this direction.