Graphics Processing Units (GPUs) are designed for massive parallelism, but their traditional memory management models are surprisingly rigid. Typically, deep learning frameworks and parallel accelerators allocate memory statically or isolate memory streams to prevent race conditions. While this works well for predictable, uniform workloads, it fails under two common conditions: irregular memory demands and multi-phase pipelines.
When memory usage fluctuates unpredictably—such as during logic simulations, data compression, or Large Language Model (LLM) key-value (KV) caching—developers must provision for the worst-case peak usage. This leads to massive underutilization and frequent Out-of-Memory (OOM) errors. To solve this, a prominent research direction focuses on breaking down these isolation boundaries, implementing dynamic, shared memory pools directly on the hardware or across the host-device boundary.
Breaking Stream Isolation with Shared Pools
Why isolate memory streams if they run on the same physical hardware? In standard deep learning setups, concurrent processes or data pipelines operate in strict silos. However, their peak memory usage rarely coincides. By unifying these pools, systems can dynamically reallocate unused memory from an idle phase to an active one.
This concept of cross-boundary sharing exploits the temporal asymmetry of memory demands. For instance, Kim et al. developed FusionFlow in FusionFlow: Accelerating Data Preparation for Machine Learning with Hybrid CPU-GPU Processing, which unifies the memory pools of data preparation and model training. Instead of letting a GPU sit with idle memory during data preprocessing, the system dynamically shifts that capacity to the training phase, allowing for larger batch sizes and preventing overflows.
Similarly, in the context of parallel data compression, Wan et al. introduced a framework called SEP in SEP: A General Lossless Compression Framework with Semantics Enhancement and Multi-Stream Pipelines that breaks default stream-level memory isolation. By allowing multiple concurrent compression pipelines to reuse the same underlying GPU memory blocks, they cut total memory usage by over 36% without sacrificing performance. Early foundational toolkits like OpenNMT, developed by Klein et al. (OpenNMT: Open-Source Toolkit for Neural Machine Translation; OpenNMT: Neural Machine Translation Toolkit), also demonstrated that external memory sharing and sharding could reduce GPU memory requirements by up to 70% during complex neural machine translation tasks.
Isolated Streams (Traditional):
[ Stream 1: Peak Memory ] [ Stream 2: Idle Memory ] -> Fixed Boundaries (OOM Risk)
Shared Memory Pool (Modern):
[ Stream 1 Pool <---> Stream 2 Pool ] -> Dynamic Reallocation (Smooths Peaks)
Autonomous On-GPU Paging and Compaction
When memory demands are highly irregular, relying on the host CPU to reallocate GPU memory introduces a devastating synchronization bottleneck. Every time a thread runs out of space, the GPU must pause, signal the CPU over the PCIe bus, wait for a new allocation, and resume execution.
To bypass this, modern techniques move memory management entirely onto the GPU hardware. Wang et al. first pioneered this for parallel logic simulations in Distributed time, conservative parallel logic simulation on GPUs by implementing a dynamic paging mechanism directly on the GPU to handle fluctuating message queues.
Building on this concept, Fang et al. eliminated CPU-GPU synchronization bottlenecks entirely in gate-level simulations (Acceleration of Timing-Aware Gate-Level Logic Simulation Through One-Pass GPU Parallelism). When a GPU thread exhausts its current memory page, it uses hardware-level atomic operations to request a new fixed-size block from a pre-allocated pool and links them via pointers. Because this happens entirely on the hardware side, memory management is decoupled from the execution topology, completely removing the CPU from the allocation loop.
A similar intuition applies to LLM inference, where the KV cache grows and fragments memory unpredictably. Zhang et al. introduced DiffKV in DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction, which uses parallel KV compaction directly on the GPU. Instead of leaving fragmented gaps in memory as tokens are processed and discarded, an on-GPU manager rapidly packs fragmented memory into contiguous regions in parallel, drastically maximizing throughput without involving the host CPU.
Virtualizing the Host-Device Boundary
When a workload simply cannot fit within the physical constraints of VRAM, the next logical step is to virtualize memory across the host (CPU) and device (GPU) boundary. Rather than treating the CPU and GPU as isolated systems connected by a slow bus, virtualized memory management treats them as a single, hierarchical address space.
Yang et al. demonstrated this with mmCNN in A Novel Memory-Scheduling Strategy for Large Convolutional Neural Network on Memory-Limited Devices, a mixed-memory approach that allows large convolutional networks to run on memory-limited devices by intelligently scheduling data transfers between host and device memory. However, naive swapping causes severe latency penalties. To mitigate this, frameworks like PoocH, developed by Ito et al. (Profiling based out-of-core hybrid method for large neural networks: poster), combine dynamic profiling with hybrid memory management. By profiling the network's memory access patterns during the first few iterations, the system creates an optimal schedule: it keeps critical tensors in VRAM, swaps predictable intermediate states to the CPU, and selectively recomputes others on the fly.
This hybrid strategy of swapping and selective recomputation is especially powerful in pipeline parallel training. As shown by Zhao et al. in vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training, memory pressure is often concentrated heavily in the initial stages of a model pipeline. Virtualizing this pipeline allows the system to automatically re-balance layers and swap tensors dynamically, enabling batch sizes nearly four times larger than static parallel systems.
Ultimately, hardware trends are moving toward making this seamless. As noted by van der Ouderaa et al. in Towards Large Scale Training on Apple Silicon, unified memory architectures natively allow the CPU and GPU to share the same physical memory pool. This structural shift completely eliminates communication overhead and opens new avenues for large-scale distributed training on consumer hardware.
Key Foundational Work
Distributed time, conservative parallel logic simulation on GPUs (2010) — contributed 3 concept elements to this direction.
OpenNMT: Open-Source Toolkit for Neural Machine Translation (2017) — contributed 1 concept elements to this direction.