In modern deep learning hardware, the primary bottleneck to performance and energy efficiency is not computation, but data movement. Performing an arithmetic operation on-chip costs orders of magnitude less energy than fetching a weight from off-chip Dynamic Random Access Memory (DRAM). This stark disparity has given rise to the "memory wall"—a barrier where accelerators spend most of their time and power budget simply shuffling data back and forth.
To bypass this wall, a major research direction focuses on keeping an entire neural network's parameters and active working memory strictly inside high-speed, on-chip storage (such as SRAM, registers, or scratchpads). However, because on-chip real estate is physically limited and expensive, researchers must employ aggressive hardware-algorithm co-design strategies to squeeze large-scale models into tiny local memory footprints.
The Performance Cliff of Off-Chip Memory
To understand why fitting models entirely on-chip is so critical, it helps to look at what happens when they fail to do so. As Hosseininoorbin et al. demonstrate in Exploring edge TPU for network intrusion detection in IoT, hardware accelerators exhibit a sharp, bi-modal "performance cliff." When a model's parameters fit comfortably within the integrated on-chip memory, the accelerator operates at peak throughput and optimal energy efficiency. The moment the model size crosses that rigid hardware boundary, the system is forced to fetch data from external storage, causing latency and power consumption to spike dramatically.
This capacity constraint represents a universal scaling limit across diverse hardware paradigms:
- GPUs: High-performance inference engines frequently assume that all model parameters reside within local caches to maintain low latency; exceeding this boundary causes performance to become entirely bounded by off-chip bandwidth, as noted by Aminabadi et al. in SHARP: An Adaptable, Energy-Efficient Accelerator for Recurrent Neural Networks.
- FPGAs: Programmable logic devices possess highly efficient but strictly finite block RAM (BRAM). Sung et al. show in Fpga Based Implementation of Deep Neural Networks Using On-chip Memory Only that navigating the trade-off between model complexity and physical BRAM capacity is the defining challenge of single-chip FPGA acceleration.
- Microcontrollers (MCUs): On ultra-low-power IoT nodes, the memory footprint is even more severe. Kwon et al. explain in YONO: Modeling Multiple Heterogeneous Neural Networks on Microcontrollers that multitasking or running modern deep learning models becomes nearly impossible when the hardware offers less than a megabyte of total storage.
- Neuromorphic Chips: Even event-based architectures like SpiNNaker 2 face strict scaling limits. Nazeer et al. demonstrate in Language Modeling on a SpiNNaker 2 Neuromorphic Chip that limited local memory per processing element restricts maximum model size, forcing developers to rely on aggressive pruning or complex multi-chip scaling.
Shrinking the Model to Fit the Cache
The most direct way to keep execution on-chip is to compress the neural network until its storage footprint drops below the hardware's memory ceiling.
In the seminal paper Learning both Weights and Connections for Efficient Neural Network, Han et al. introduced a three-step pipeline—Train, Prune, and Retrain—that eliminates redundant connections by an order of magnitude without losing accuracy. By removing zero-weight parameters, the authors made it possible to fit massive convolutional layers into local SRAM.
Mao et al. extended this concept in Deep Compression: Compressing Deep Neural Networks with Pruning by combining pruning with trained quantization and Huffman coding. This multi-stage compression allowed custom hardware architectures, such as the Efficient Inference Engine (EIE) proposed by Han et al. in EIE: Efficient Inference Engine on Compressed Deep Neural Network, to hold compressed networks entirely in on-chip SRAM, achieving massive speedups and energy reductions by bypassing DRAM entirely.
Uncompressed Model (Hundreds of MBs) ──> Exceeds On-Chip SRAM ──> DRAM Bottleneck
│
[ Pruning & Quantization ]
▼
Compressed Model (Few MBs) ───────────> Fits Inside On-Chip SRAM ──> High-Speed Compute
However, unstructured pruning creates irregular, sparse memory access patterns that can be difficult for hardware accelerators to parallelize efficiently. To solve this, researchers turned to structured compression techniques. Ding et al. (C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs) and Li et al. (Efficient Recurrent Neural Networks using Structured Matrices in FPGAs) utilized block-circulant matrices to compress Recurrent Neural Networks (RNNs). This mathematical framing reduces computational complexity from $O(k^2)$ to $O(k \log k)$ via Fast Fourier Transforms (FFT), drastically lowering the on-chip memory required to store weight matrices while maintaining high hardware throughput.
Persistent Weights and Stationary Dataflows
Beyond shrinking the model, efficiency can be gained by changing how data moves through the processor. In standard execution models, weights are streamed from memory into registers, multiplied by inputs, and then discarded. For architectures like RNNs or Transformers, where the same weights are reused sequentially across multiple timesteps, this constant reloading is highly inefficient.
To address this, Diamos et al. proposed the concept of "persistent kernels" in Persistent RNNs: Stashing Recurrent Weights On-Chip. Instead of streaming weights, the authors pinned the recurrent weights directly inside the GPU's on-chip registers across the entire duration of the execution sequence. Zhu et al. combined this persistent approach with structural pruning in Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip, allowing models five times larger to remain resident in high-speed registers.
The catch to weight-stationary dataflows is that they introduce a hard physical ceiling. As Zhu et al. and İslamoğlu et al. (ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers) point out, if the hidden layer or projection space of a network exceeds the aggregate capacity of the local register file or weight buffers, the system must revert to slower memory hierarchies, immediately destroying the latency benefits of the persistent execution flow.
Managing the Activation and Working Memory Bottleneck
Compressing the static weights of a network solves only half the problem. As a model executes, it generates intermediate feature maps—known as activations—that must also be stored in working memory.
Igor Fedorov highlights this critical distinction in SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers. Simply reducing the number of model parameters is insufficient for deployment on tiny microcontrollers because it ignores the peak RAM consumed by intermediate layers during execution. Even if a model's compressed weights fit on a chip, the network will crash if the activation tensor of a single large convolutional layer exceeds the available SRAM. Igor Fedorov demonstrates that Neural Architecture Search (NAS) frameworks must explicitly co-optimize for both parameter footprint and peak runtime activation memory.
When a model's working memory requirements inevitably outgrow local SRAM boundaries, hardware engines must orchestrate intelligent data movement to mitigate the latency penalty. Hare et al. address this in TinyOps: ImageNet Scale Deep Learning on Microcontrollers by implementing a "ping-pong" buffering scheme powered by Direct Memory Access (DMA).
[ External Memory ] ──(DMA Prefetch)──> [ SRAM Buffer A ] ──> [ Compute Core ]
│
[ External Memory ] <──(DMA Flush)──── [ SRAM Buffer B ] <───────────┘
While the compute core processes data residing in SRAM Buffer A, the DMA engine concurrently flushes completed activations from SRAM Buffer B and pre-fetches the next layer's data from high-latency external memory. This pipelining hides the latency of external transfers, allowing microcontrollers to execute larger networks without stalling the processor.
Alternative On-Chip Storage Paradigms
Because standard 6T-SRAM cells occupy a significant amount of physical silicon area, scaling up on-chip SRAM capacity quickly becomes cost-prohibitive. To circumvent this area overhead, researchers are exploring alternative on-chip storage paradigms.
One approach involves replacing area-heavy SRAM with denser, non-volatile memory structures for in-memory computing. Chen et al. propose this strategy in YOLoC: DeploY Large-Scale Neural Network by ROM-based Computing-in-Memory using ResiduaL Branch on a Chip. They utilize ultra-dense Read-Only Memory (ROM) to store the fixed, heavy "trunk" of a neural network's weights on-chip, pairing it with a small, flexible SRAM "residual branch" to handle dynamic updates. This hybrid architecture bypasses the SRAM area bottleneck, allowing larger models to fit entirely on a single die.
Another radical departure from traditional matrix multiplication is replacing neural layers entirely with memory lookups. Huang et al. demonstrate this concept in Hundred-Kilobyte Lookup Tables for Efficient Single-Image Super-Resolution. By pre-computing layer outputs and storing them in ultra-compact lookup tables (~100KB), the system maps inputs directly to outputs via indexing. These tiny tables fit entirely within local L1/L2 caches, bypassing the need for large weight matrices and expensive floating-point hardware altogether.
Key Foundational Work
Language Identification and Modeling in Specialized Hardware (2015) — contributed 2 concept elements to this direction.
Learning both Weights and Connections for Efficient Neural Network (2015) — contributed 1 concept elements to this direction.
A Massively-Parallel {SIMD} Processor for Neural Network and Machine Vision Applications (1993) — contributed 1 concept elements to this direction.