As deep learning models scale toward trillions of parameters, they have collided with a physical boundary known as the "memory wall." While GPU compute power has grown significantly, the high-bandwidth memory (VRAM) on these accelerators remains a scarce and expensive resource. A 175B parameter model requires hundreds of gigabytes just to store its weights, far exceeding the capacity of even the most advanced single GPUs.
To bypass this limit, researchers are reimagining the relationship between processing and storage. Instead of treating GPU memory as a static container for the entire model, they are developing hierarchical memory systems that treat VRAM as a high-speed cache within a multi-tier architecture spanning CPU RAM and even persistent SSD storage.
The Anatomy of the Memory Wall
In a typical training setup, the GPU must hold several types of data: model weights, gradients, activations, and optimizer states. As Julia Gusak and colleagues note in their survey on efficient training, optimizer states (like those in Adam) are often the primary culprits, consuming several times more memory than the weights themselves.
When these requirements exceed VRAM, the traditional response was to add more GPUs. However, this "compute arms race" is often inaccessible to academic labs. Manuel R. Ciosici and Leon Derczynski demonstrate in Training a T5 Using Lab-sized Resources that by offloading these memory-intensive optimizer states to system RAM, researchers can train large-scale models on a single workstation that would otherwise require a massive cluster.
Offloading and Asynchronous Streaming
The core mechanism of tiered memory is offloading: moving tensors from the GPU to the larger, slower CPU memory (DRAM) and bringing them back only when needed for computation.
The challenge is that CPU-GPU communication is much slower than internal GPU memory access. To prevent the processor from sitting idle while waiting for data, systems like AxoNN, developed by Siddharth Singh and Abhinav Bhatele, use asynchronous, message-driven execution. By retrieving data in small "buckets" and overlapping these transfers with active computation, they can reduce the GPU memory footprint by 4x with minimal performance loss.
This concept extends to even slower storage tiers. In Merlin HugeCTR, Zehuan Wang and his team describe a three-tier hierarchy for recommendation systems:
- GPU VRAM: Stores the most frequently accessed "hot" features.
- System RAM: Holds "warm" data.
- SSD: Acts as a persistent store for the "long-tail" of infrequent embeddings.
GPU as a Dynamic Cache
A significant shift in this research direction is treating GPU memory not as a fixed residence for parameters, but as a dynamic cache.
In the domain of 3D computer vision, Chonghao Zhong and colleagues introduced TideGS, a system that scales 3D Gaussian Splatting to over a billion primitives. As detailed in TideGS: Scalable Training via Out-of-Core Optimization, the system only loads the "active working set" required for a specific training iteration into the GPU. This allows them to process models that are an order of magnitude larger than the physical VRAM.
Similarly, for Large Language Models (LLMs), Jie Peng and Hua Qu developed M2Cache. Their framework harnesses DRAM and SSDs to run massive LLMs on consumer-grade GPUs, using multi-level caching to hide the latency of loading weights from slower storage.
Disentangling Reasoning from Memorization
Beyond just solving hardware constraints, hierarchical memory enables new architectural paradigms. David Grangier and Hadi Pouransari propose separating a model into two parts: a small "anchor" model for general reasoning and a massive "hierarchical memory bank" for world facts.
By fetching only a tiny, context-relevant slice of this memory for each prompt, they achieve the performance of much larger models. This approach mirrors how humans use external references: we don't memorize every fact in an encyclopedia; we maintain the reasoning skills to look up the specific information we need. This pretraining-with-memories strategy allows massive knowledge bases to be deployed on memory-constrained edge devices by keeping only the "reasoning" core in fast memory.
Democratizing Large-Scale AI
The ultimate goal of these tiered systems is democratization. By breaking the VRAM wall, researchers are making it possible to train and run state-of-the-art models without datacenter-scale resources.
Frameworks like OpenBMB, described by Guoyang Zeng and Zhiyuan Liu, provide toolkits for tiered memory parameter streaming, enabling 10-billion-parameter models to run on consumer GPUs. Similarly, the LLMQ framework by Erik Schultheis and Dan Alistarh combines aggressive offloading with lower-precision training to achieve datacenter-level efficiency on affordable hardware like the RTX 4090.
Through these hierarchical innovations, the field is moving away from a world where model size is strictly limited by GPU memory, toward one where the entire storage capacity of a system can be harnessed for intelligence.