Hiding Memory Latency with Asynchronous Software Pipelining

This direction is split into other directions: latency hiding (ping-pong buffering), latency hiding (asynchronous data transfer). Its last version stays readable and citable here.

Version 1 of 1523 papers665 conceptsfrozen 2026-10-04

In modern high-performance computing, the primary bottleneck is rarely the speed of the processor itself, but rather the speed at which data can be moved into that processor. Whether training a Large Language Model (LLM) or processing genomic sequences, hardware accelerators like GPUs often sit idle, waiting for data to arrive from memory.

The research direction across these 315 papers focuses on asynchronous software pipelining: a set of techniques designed to "hide" this memory latency by overlapping data movement with active computation. The goal is to ensure that by the time a processor finishes its current task, the data for the next task is already waiting in its local registers or cache.

The Intuition: Decoupling Producers and Consumers

To understand why this works, imagine a professional kitchen. If a chef (the compute core) has to stop cooking to walk to the pantry (main memory) every time they need an ingredient, the kitchen is inefficient. Instead, a sous-chef (the DMA or memory controller) can act as a "producer," fetching ingredients in the background while the chef—the "consumer"—stays at the stove.

In FlashAttention-3, Jay Shah and Tri Dao apply this by specializing hardware threads into producer and consumer roles. By using asynchronous instructions on NVIDIA’s Hopper architecture, they allow the GPU to fetch the next tile of an attention matrix while the Tensor Cores are still multiplying the current one. This decoupling prevents the "stalls" that typically occur when a processor hits a memory wall.

Double Buffering and Latency Masking

The most common mechanism for achieving this overlap is double buffering. The system allocates two memory buffers: while the processor computes on Buffer A, the memory controller fills Buffer B. Once the computation is done, the roles swap.

This technique is foundational across architectures. Philip S. Yu and colleagues demonstrated this on the IBM Cell processor in CellSort, using asynchronous Direct Memory Access (DMA) to pre-fetch data blocks. Similarly, in the context of many-core supercomputers, Changxi Liu developed swTVM, a compiler that automatically inserts DMA instructions to hide latency on the Sunway processor. The intuition remains the same: if the computation time is greater than or equal to the transfer time, the cost of moving data effectively drops to zero.

Scaling Beyond GPU Memory

As models grow, they often exceed the physical memory of a single GPU. Researchers have extended asynchronous pipelining to manage data movement between the GPU, system RAM, and even SSDs.

  • Activation Offloading: During LLM training, Kai Wu and the team behind SSDTrain offload intermediate tensors to NVMe SSDs. By perfectly overlapping these transfers with the forward pass, they reduce memory pressure without slowing down training.
  • KV Cache Swapping: For long-context inference, Hanshi Sun introduced ShadowKV, which stores the "Value" cache in host memory. The system retrieves these values just-in-time, overlapping the retrieval with the reconstruction of "Key" projections on the GPU.
  • Dynamic Swapping: Bingsheng He and Sai Ho Yeung developed AutoSwap, which uses independent execution streams to move data between CPU and GPU. This allows for larger batch sizes by treating the CPU RAM as an extension of the GPU's memory, with the latency hidden behind active kernels.

Architectural and Compiler Automation

While manual pipelining is powerful, it is difficult to implement. Recent work focuses on making these optimizations automatic. Shaoyi Cheng proposed a High-Level Synthesis (HLS) flow that maps C++ code to a dataflow template, automatically decoupling memory accesses from computation.

In the realm of Processing-in-Memory (PIM), Hyucksung Kwon and colleagues developed LoL-PIM, which uses out-of-order command scheduling. By treating memory commands as dynamic instructions, the system can reorder them to fill pipeline bubbles, ensuring that the internal buses of the memory chips are never the bottleneck.

Summary of the Impact

The shift toward asynchrony represents a fundamental change in how we view program execution. Instead of a linear sequence of "fetch then compute," modern high-performance systems operate as a continuous flow. By treating data movement as a first-class citizen that runs in parallel with math, researchers like Ganesh Bikshandi and Bertil Schmidt have shown that we can achieve near-theoretical peak performance even on workloads that are traditionally considered "memory-bound."

Papers in this version

Showing 60 of 523 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hiding_memory_latency_with_asynchronous__a8d3bc1f,
  title = {Hiding Memory Latency with Asynchronous Software Pipelining},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hiding-memory-latency-with-asynchronous-software-pipelining/rdr_a8d3bc1f3de0463aad940c99b17e2943},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_hiding_memory_latency_with_asynchronous__a8d3bc1f_v1,
  title = {Hiding Memory Latency with Asynchronous Software Pipelining},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hiding-memory-latency-with-asynchronous-software-pipelining/rdr_a8d3bc1f3de0463aad940c99b17e2943/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}