Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap

Shagnik Pal, Shaizeen Aga, Suchita Pati, Mahzabeen Islam, Lizy K. JohnarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) like Llama 3 and DeepSeek scale to thousands of GPUs, the "communication tax"—the time spent moving data between chips—becomes a massive performance bottleneck. Standard parallelization techniques (like Tensor or Data Parallelism) shard models and inputs across GPUs. To hide the latency of moving these shards, developers use "overlap," where one shard is communicated while another is being computed. However, this "shard-level" overlap is often too coarse and frequently relies on peer-to-peer (P2P) communication patterns that leave modern, high-bandwidth direct-connect topologies underutilized. Furthermore, running computation and communication kernels simultaneously on the same GPU often leads to resource contention, where tasks compete for cache and memory bandwidth.

FiCCO: Finer-Grain Compute Communication Overlap

Researchers from AMD and UT Austin have introduced FiCCO, a technique that decomposes distributed machine learning tasks into even smaller "finer-grained" chunks than standard methods. Instead of sharding data just by the number of GPUs, FiCCO shards one level deeper (e.g., eight chunks for an eight-GPU system). This unlocks "All-to-All" communication patterns that keep all network links busy.

Comparison of Serial Execution vs. FiCCO Execution showing finer-grain decomposition and overlap.

FiCCO explores a complex design space for these schedules, including sharding dimensions (1D row-based vs. 2D column-based), kernel fusion strategies, and step uniformity. Because no single schedule wins every time, the authors developed a decision tree based on GEMM dimensions and hardware-specific limits like the "Op-to-Byte" (OTB) ratio.

Addressing Interference with DMA Offloading

Traditionally, GPU cores manage communication. FiCCO offloads this to dedicated DMA (Direct Memory Access) engines. This eliminates "compute interference" and reduces cache contention, leaving the GPU cores free to focus entirely on mathematical operations.

Evolution from baseline execution to shard-level overlap and the impact of DMA on reducing interference.

The paper highlights two primary types of loss: Decomposition Inefficiency Loss (DIL), where smaller kernels are inherently less efficient at saturating GPU units, and Contention Inefficiency Loss (CIL), where overlapping tasks fight for resources. By moving communication off the GPU cores, FiCCO significantly lowers CIL. The authors found that DMA-based communication causes far lower slowdowns for the primary computation than traditional GPU-core-driven libraries like RCCL.

Key Results and Insights

The study evaluated FiCCO using real-world GEMM kernels from state-of-the-art LLMs, including Llama-2-70b, Llama-3-405b, Mixtral, and DeepSeek.

  • Performance: FiCCO delivers up to a 1.6x speedup over serial execution and significantly outperforms standard shard-level overlap.
  • Heuristic Accuracy: The proposed heuristic correctly identifies the optimal execution schedule in 81% of unseen scenarios.
  • Contention Reduction: Using DMA engines reduced contention-related slowdowns from ~11% (with standard core-driven communication) to nearly negligible levels in many cases.
  • Topology Utilization: On direct-connect meshes, standard P2P patterns can be up to 3.9x slower than serial execution because they leave most links idle. FiCCO’s finer granularity allows for All-to-All patterns that saturate the mesh.

Conclusion

FiCCO demonstrates that moving from coarse shard-level overlap to finer-grained, DMA-offloaded communication can significantly improve the efficiency of distributed ML training. While current hardware limitations exist—such as the lack of native 2D memory copy support in GPU DMA engines—the results suggest that future hardware and compiler integrations (e.g., into Triton or PyTorch) could further unlock the potential of high-bandwidth interconnects.

Summary updated

BibTeX
@misc{pal2025design,
  author = {Shagnik Pal and Shaizeen Aga and Suchita Pati and Mahzabeen Islam and Lizy K. John},
  title = {{Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap}},
  year = {2025},
  eprint = {2512.10236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.10236},
}
Figure