When optimizing deep learning models, standard practice often focuses on reducing the number of floating-point operations (FLOPs). However, in modern hardware architectures, arithmetic computation is remarkably cheap; the real bottleneck is data movement. Fetching a single weight from external memory (such as DRAM or Flash) can consume orders of magnitude more energy and introduce significantly higher latency than the actual mathematical operation itself. This performance and energy bottleneck is widely known as the memory wall.
To bypass the memory wall entirely, a prominent research direction focuses on complete on-chip memory residency. The core intuition is simple: if a neural network's weights, intermediate activations, and state queues can be compressed or structured to fit entirely within a processor's local, high-speed memory (such as SRAM, registers, or last-level cache), the energy tax of off-chip data movement drops to zero. This architectural alignment transforms memory-bound bottlenecks into compute-bound pipelines, unlocking massive efficiency gains for edge devices, wearables, and specialized accelerators.
Compressing Parameters to Fit On-Chip Cache
The primary challenge of on-chip execution is the capacity mismatch between large deep learning models and tiny on-chip memory buffers. To bridge this gap, researchers employ aggressive compression pipelines combining pruning, quantization, and alternative structural representations.
In a foundational paper, Han et al. introduced a method for Learning both Weights and Connections for Efficient Neural Network, demonstrating that pruning redundant connections allows massive networks to shrink by an order of magnitude without losing accuracy. This concept was expanded by Mao et al. in Deep Compression: Compressing Deep Neural Networks with Pruning, which combined pruning, trained quantization, and Huffman coding to reduce model sizes up to 49x. By shrinking a model down to a fraction of its original size, the entire parameter set can reside within high-speed on-chip SRAM cache, eliminating the high energy cost of off-chip retrieval.
[Traditional Inference]
DRAM (High Energy Tax) ──> On-Chip Cache ──> Processing Units (ALUs)
[SRAM-Resident Inference]
On-Chip SRAM (Compressed Model) ───────────> Processing Units (ALUs) ==> Bypasses DRAM Wall
Beyond standard pruning, extreme quantization pushes models into low-bit configurations. Sung et al. demonstrated that 2-bit and 3-bit quantization schemes allow recurrent and deep networks to run entirely within low-power on-chip cache or block RAM, providing the necessary efficiency for continuous real-time processing on wearable electronics (Dynamic Hand Gesture Recognition for Wearable Devices with Low Complexity Recurrent Neural Networks; Fpga Based Implementation of Deep Neural Networks Using On-chip Memory Only).
Single-author works have reinforced that the primary benefit of these compressed representations is memory localization rather than raw FLOP reduction. For instance, Akhauri developed flexible bit-wise representations to minimize data movement energy between memory and the processor (HadaNets: Flexible Quantization Strategies for Neural Networks), while Ahmed designed an encoder-decoder compression scheme for Vision Transformers, noting that prioritizing parameter placement within on-chip memory drives the most significant energy savings at the edge (DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge).
For highly specialized domains like recommendation systems, where embedding tables routinely span gigabytes, Jha et al. replaced traditional tables with a dual-Bloom filter encoding scheme in Mem-Rec: Memory Efficient Recommendation System using Alternative Representation. This alternative representation compressed the model sufficiently to fit within a CPU's last-level cache, bypassing the memory bandwidth bottleneck during inference.
Taming the Activation and State Footprint
Fitting a model's weights on-chip solves only half the problem. During inference, intermediate activations and sequential state queues can expand rapidly, causing a "memory explosion" that forces data to spill back into off-chip storage. Complete residency requires managing both static parameters and dynamic execution states.
One approach is to size hardware structures and schedule operations around the network's peak activation demands. Al Bahou et al. designed a binary convolutional neural network accelerator where the internal SRAM is sized specifically to accommodate the largest feature map of a target layer, ensuring all intermediate data transfers remain strictly local (XNORBIN: A 95 TOp/s/W hardware accelerator for binary convolutional neural networks). Similarly, Venkatesh et al. restructured recurrent cells and layer scheduling to guarantee that active weight matrices never exceed local buffer capacities (Memory-efficient Speech Recognition on Smart Devices).
[Standard Layer Execution]
Layer N Activations ──(Spill to Off-Chip Storage)──> Layer N+1 Input
[SRAM-Constrained Execution (e.g., Raster-Scanning)]
Row-by-Row Processing ──(Buffered Locally in SRAM)──> Immediate Consumption
Alternative execution paradigms can also fundamentally alter memory dynamics. In RaScaNet: Learning Tiny Models by Raster-Scanning Images, Yoo et al. introduced a row-by-row raster-scanning approach for vision tasks. By processing images sequentially rather than buffering entire feature maps, they eliminated the massive activation memory overhead typical of early convolutional layers, fitting high-performance vision models into less than 60KB of SRAM.
For autoregressive and sequential tasks, the performance bottleneck is often the management of internal states rather than raw computational throughput. Kastner demonstrated that keeping the entire model state and its associated convolutional queues within on-chip memory blocks allows specialized FPGA hardware to drastically outperform high-end GPUs by avoiding off-chip latency (FastWave: Accelerating Autoregressive Convolutional Neural Networks on FPGA). Even during on-device training, memory footprints can be constrained; Cai et al. showed that freezing weights and updating only biases reduces intermediate activation storage enough to let the entire learning process remain within a CPU's L3 cache (TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning).
Hardware Co-Design and Distributed SRAM
When a model cannot be compressed enough to fit into a single standard SRAM block, researchers turn to hardware-algorithm co-design or distributed topologies to maintain on-chip residency.
One co-design strategy involves utilizing alternative on-chip storage mediums. Chen et al. developed YOLoC: DeploY Large-Scale Neural Network by ROM-based Computing-in-Memory using ResiduaL Branch on a Chip, which pairs an ultra-dense, on-chip Read-Only Memory (ROM) trunk to store fixed weights with a small, flexible SRAM residual branch. This architecture eliminates external DRAM weight fetching entirely, ensuring all computational operands are available locally within the processing-in-memory fabric.
[Distributed On-Chip Architecture]
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Microcontroller1 │ │ Microcontroller2 │ │ Microcontroller3 │
│ ┌──────────────┐ │ │ ┌──────────────┐ │ │ ┌──────────────┐ │
│ │SRAM (Layer 1)│ │ ───> │ │SRAM (Layer 2)│ │ ───> │ │SRAM (Layer 3)│ │
│ └──────────────┘ │ │ └──────────────┘ │ │ └──────────────┘ │
└──────────────────┘ └──────────────────┘ └──────────────────┘
└─────────────────────────────── Distributed Model ──────────────────────────────┘
Alternatively, the memory workload can be distributed horizontally across multiple chips. Bochem et al. proposed a distributed inference strategy for Small Language Models in Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs. By partitioning Transformer layers across a network of low-power microcontrollers without duplicating weights, they ensured that each sub-network fit entirely within its respective microcontroller's on-chip SRAM, bypassing the memory wall through localized, cooperative execution.
While complete on-chip residency remains the gold standard for efficiency—as seen in DRAM-less accelerators designed by Sun et al. and Xiang et al.—some frameworks introduce hybrid approaches. For instance, Yu et al. automated weight streaming between on-chip and off-chip memory to maximize performance when hardware constraints are absolute (Auto WS: Automate Weights Streaming in Layer-Wise Pipelined DNN Accelerators).
However, for continuous, power-constrained applications like industrial package monitoring (Renault et al.) or always-on speech recognition (Li et al.), the consensus remains clear: compressing frequently invoked model components until they fit entirely within local SRAM is the most effective way to eliminate the off-chip data movement tax and achieve optimal energy efficiency.
Key Foundational Work
Fpga Based Implementation of Deep Neural Networks Using On-chip Memory Only (2016) — contributed 2 concept elements to this direction.
Learning both Weights and Connections for Efficient Neural Network (2015) — contributed 1 concept elements to this direction.
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2016) — contributed 1 concept elements to this direction.