Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs

Gengyu Rao, Ghadeer Ahmed H Alabandi, Su Ann Chong, Thomas Huber, Kailash Gogineni, Subhajit Dutta Chowdhury, Saleel Kudchadker, Suchita Pati, Zirui Zhang, Zhen Huang, Yanyuan Qin, Xiaoming Peng, Lihuan Zhang, Botao Hu, Wei Huang, Yang Bo, Wen Xie, Xiangyu Zhao, Ruibin Zhang, Xiaobo Chen, Vidushi Goyal, Anshu Raina, Claire Songhyun Lee, Jiakun Yan, Yao Fu, Nusrat Islam, Ravi Dwivedula, Jie Zhang, Maria Garzaran, Zhenyu Gu, Emad BarsoumarXiv 2026 · arXiv

TL;DR

AMD researchers have unveiled a hierarchical data pipelining strategy that significantly boosts LLM training efficiency by offloading data movement to specialized hardware. By utilizing features like the Tensor Data Mover (TDM) and dedicated DMA engines, they achieved up to 1.88× speedup in kernels and 1.33× faster communication-computation overlap, effectively keeping GPU compute units busy rather than waiting for data.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale, the primary bottleneck isn't just raw arithmetic power—it's the "data-to-compute" gap. Even the most powerful GPUs sit idle if the data they need is stuck in slow off-chip memory or traversing a network from another GPU.

Traditionally, GPUs hide memory latency by "oversubscribing" threads (scheduling more work than the hardware can run at once). However, in modern LLM training—especially with complex algorithms like FlashAttention or the irregular patterns of Sparse Attention—this approach hits a wall. Moving data often competes with the actual math for the same execution resources (Compute Units). The authors aim to transform data movement from a competitive overhead into a background process that "feeds" the compute units a steady stream of data.

Methods

The paper introduces a hierarchical approach using three specific AMD hardware features:

  1. Tensor Data Mover (TDM): A hardware DMA engine that moves tensor data between High-Bandwidth Memory (HBM) and on-chip Local Data Share (LDS). Unlike standard loads, TDM is "tensor-aware," meaning it can handle padding and specific tensor dimensions automatically.
  2. Transpose-Load ISA: Specialized instructions (like ds_load_tr16_b128) that perform matrix transposition during the load from on-chip memory to registers. This eliminates the need for expensive manual register shuffles or strided memory accesses.
  3. SDMA and RDMA: The authors utilize System Direct Memory Access (SDMA) for scale-up (intra-node) and Remote DMA (RDMA) for scale-out (inter-node) communication. By offloading these to dedicated engines, they free up the Compute Units (CUs) to continue performing math during communication.
Figure 1: Simplified memory hierarchy of AMD CDNA5 GPU [4].

To make these features accessible, the team integrated them into the Primus and Primus-Turbo libraries, providing high-level PyTorch APIs for developers.

Datasets

The researchers evaluated their implementation using several industry-standard models and benchmarks:

  • LLM Architectures: Llama 3.1 (8B and 405B shapes) and GPTOSS 20B.
  • Diffusion Models: FLUX.1-Schnell.
  • Case Study: Compressed Sparse Attention (CSA) as used in the DeepSeek V4 architecture, which represents a highly challenging data movement pattern.
  • Benchmarks: Selected datasets from the MLPerf suite.

Key Insights and Ideas

  • The Register Pressure Win: One of the most significant "aha" moments was that TDM doesn't just hide latency; it changes the resource math. Without TDM, kernels need high occupancy (more threads) to hide latency, which limits each thread to fewer registers. With TDM, the kernel can drop to one "wave" per SIMD, doubling the available registers (up to 1024 VGPRs). This allows for much deeper software pipelining and better data reuse.
  • Hardware-Algorithm Alignment: The TDM’s sparse gather support perfectly matches the "top-k" indexing used in modern sparse attention. A single TDM instruction can replace a complex loop of address calculations and manual loads.
  • Power and Frequency Stability: The authors noted that traditional communication methods trigger unnecessary data movement across chiplets, causing power competition. Using dedicated SDMA engines reduced these power spikes and prevented the compute-die frequency from dropping during transfers.
Table 1: ROCm and CUDA GPU programming terminology [4, 21].

Key Results

  • Kernel Speedups: TDM-enabled kernels achieved a geometric mean speedup of 1.88× (reaching up to 2.74× in some cases). The Transpose-load ISA provided an additional 1.19× average speedup by streamlining the data layout for math operations.
  • Communication Overlap: Using SDMA for AllGather and ReduceScatter operations reduced the performance "regression" (slowdown) from ~30% down to ~10% compared to traditional Compute Unit-based communication.
  • Scale-out Performance: On a 32-GPU cluster, the hierarchical SDMA/RDMA approach made overlapped GEMM (General Matrix Multiply) operations 1.33× faster than conventional methods.
Table 5: matrix transpose with / without LDS transpose load.

Limitations and Future Work

  • Platform Maturity: The MI455 results were collected on a "bring-up platform," meaning these are early snapshots rather than the final performance ceiling of the hardware.
  • Software Integration: While integrated into Primus, wider adoption will depend on these features being fully transparent in mainstream frameworks like standard PyTorch or JAX.
  • Future Directions: The team is exploring "reverse offloading," which uses host CPU proxy threads to initiate network operations, potentially further reducing GPU overhead. They also plan to integrate these features into specialized kernels like DeepEP and MegaMoE.

Summary updated

Abstract

Large language models (LLM) have been the key focus of recent machine learning breakthroughs. As the model size grows exponentially, the computation needed for training is astronomical. A GPU can do a huge amount of arithmetic. However, the data its compute units need is not always available when they are ready. This data may still be in slow memory or on another GPU. While the chip waits, expensive compute sits idle, and at scale even a small drop in utilization translates into substantial cost. In this paper, we treat this waiting time as the primary bottleneck. We focus on hardware features designed for explicit data movement. These features load data from off-chip memory onto the chip, rearrange it into the layout the computation requires as it is loaded, and transfer it between GPUs. By orchestrating them, we keep a steady pipeline of data flowing to the compute units so that they remain busy. We present kernel and framework implementations that address this bottleneck using three key data-movement instructions and hardware features. The Tensor Data Mover (TDM) moves tensor data from high-bandwidth memory (HBM) into on-chip memory. Transpose-load instructions transpose tensor data into the layout that LLM computations require as it is loaded on-chip. Finally, System DMA (SDMA) and Remote DMA (RDMA) move data between GPUs over the scale-up and scale-out networks, respectively. Our results show that TDM and transpose-load provide on average 1.88x and 1.19x speedup at the kernel level, respectively. The SDMA and RDMA lead to on average 1.16x and 1.33x speedup for computation overlapped with communication. This is the first AMD LLM training system to utilize all the GPU data pipelining features. Our implementation has been integrated into open source framework and library for AMD GPUs. https://github.com/AMD-AGI/Primus-Turbo https://github.com/AMD-AGI/Primus
BibTeX
@misc{rao2026scheduling,
  author = {Gengyu Rao and Ghadeer Ahmed H Alabandi and Su Ann Chong and Thomas Huber and Kailash Gogineni and Subhajit Dutta Chowdhury and Saleel Kudchadker and Suchita Pati and Zirui Zhang and Zhen Huang and Yanyuan Qin and Xiaoming Peng and Lihuan Zhang and Botao Hu and Wei Huang and Yang Bo and Wen Xie and Xiangyu Zhao and Ruibin Zhang and Xiaobo Chen and Vidushi Goyal and Anshu Raina and Claire Songhyun Lee and Jiakun Yan and Yao Fu and Nusrat Islam and Ravi Dwivedula and Jie Zhang and Maria Garzaran and Zhenyu Gu and Emad Barsoum},
  title = {{Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs}},
  year = {2026},
  eprint = {2611.05751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05751},
}
Figure