The fundamental bottleneck in modern AI is not computation, but movement. While processors can perform trillions of operations per second, the energy and time required to fetch weights from off-chip memory (DRAM) are orders of magnitude higher than the cost of the math itself. This "memory wall" is particularly punishing for real-time applications like speech recognition or high-frequency trading, where every microsecond of latency matters.
To break this wall, researchers have converged on a strategy of on-chip weight persistence: keeping a model’s entire parameter set within the high-speed, local memory of the processor (registers or SRAM) throughout the entire execution.
The Persistent Kernel Mechanism
In a standard execution model, a processor loads a layer's weights, processes the data, and then discards those weights to make room for the next layer. For Recurrent Neural Networks (RNNs), this happens at every single timestep.
Greg Diamos and colleagues at Baidu pioneered the Persistent RNN approach, which fundamentally changes this lifecycle. Instead of reloading weights, they "stash" them in the GPU’s on-chip registers. By keeping the weights resident across multiple timesteps, they achieved a 30x speedup for small batch sizes. The intuition is simple: if the weights never leave the chip, the memory bandwidth bottleneck disappears, and the system becomes limited only by the raw speed of the arithmetic units.
Fitting the Model: The Compression Necessity
The primary challenge of on-chip persistence is the "hard ceiling" of physical space. On-chip memory is incredibly fast but extremely scarce—typically measured in megabytes, while modern models are measured in gigabytes.
To bridge this gap, researchers use hardware-algorithm co-design. Song Han introduced Deep Compression, a pipeline of pruning and quantization that shrinks models like VGG-16 by up to 49x. This allows complex networks to fit entirely within on-chip cache, enabling high-efficiency deployment on mobile devices. Similarly, Zhe Li demonstrated that using structured matrices can reduce the memory footprint of RNNs enough to fit them into the limited Block RAM of FPGAs, bypassing the "memory wall" entirely.
Beyond Weights: Single-Load and Fused Execution
Persistence isn't just about weights; it’s also about intermediate activations. In standard deep learning, the output of one layer is written to DRAM and then read back for the next layer.
Recent work like ME-ViT by Kyle Marino enforces a single-load policy, where data is loaded once and kept on-chip for the entire duration of the Vision Transformer's execution. This is mirrored in GPU software by ClusterFusion, where Xinhao Luo and others fuse multiple LLM decoding stages into a single kernel. By keeping intermediate data in shared memory, they avoid the latency of global memory synchronization.
The Performance Cliff
The most critical insight across this research is the non-linear nature of performance. As noted in studies of the Google Edge TPU by Seyedehfaezeh Hosseininoorbin, there is a "performance cliff."
As long as the model fits on-chip, energy efficiency and throughput remain high. However, the moment a model’s parameters or activations exceed the capacity of the on-chip SRAM, the system is forced to "spill" data to off-chip DRAM. This transition causes a massive, non-linear spike in energy consumption and latency. This constraint defines the design of systems like IBM’s NorthPole, which prioritizes on-chip weight storage to achieve extreme efficiency, but at the cost of a rigid upper bound on model size.
Scaling to the Wafer Level
To support the massive parameter counts of Large Language Models (LLMs) while maintaining persistence, the hardware itself must grow. Yeqi Huang and the team behind WaferLLM explore using wafer-scale accelerators (like the Cerebras WSE). By treating an entire silicon wafer as a single chip with hundreds of megabytes of distributed SRAM, they can keep billions of parameters resident on-chip, achieving 10–20× speedups over traditional GPU clusters that must constantly swap data over slower interconnects.