Introduction
As Large Language Models (LLMs) like Llama 3 and DeepSeek scale to thousands of GPUs, the "communication tax"—the time spent moving data between chips—becomes a massive performance bottleneck. Standard parallelization techniques (like Tensor or Data Parallelism) shard models and inputs across GPUs. To hide the latency of moving these shards, developers use "overlap," where one shard is communicated while another is being computed. However, this "shard-level" overlap is often too coarse and frequently relies on peer-to-peer (P2P) communication patterns that leave modern, high-bandwidth direct-connect topologies underutilized. Furthermore, running computation and communication kernels simultaneously on the same GPU often leads to resource contention, where tasks compete for cache and memory bandwidth.
FiCCO: Finer-Grain Compute Communication Overlap
Researchers from AMD and UT Austin have introduced FiCCO, a technique that decomposes distributed machine learning tasks into even smaller "finer-grained" chunks than standard methods. Instead of sharding data just by the number of GPUs, FiCCO shards one level deeper (e.g., eight chunks for an eight-GPU system). This unlocks "All-to-All" communication patterns that keep all network links busy.
FiCCO explores a complex design space for these schedules, including sharding dimensions (1D row-based vs. 2D column-based), kernel fusion strategies, and step uniformity. Because no single schedule wins every time, the authors developed a decision tree based on GEMM dimensions and hardware-specific limits like the "Op-to-Byte" (OTB) ratio.
Addressing Interference with DMA Offloading
Traditionally, GPU cores manage communication. FiCCO offloads this to dedicated DMA (Direct Memory Access) engines. This eliminates "compute interference" and reduces cache contention, leaving the GPU cores free to focus entirely on mathematical operations.
The paper highlights two primary types of loss: Decomposition Inefficiency Loss (DIL), where smaller kernels are inherently less efficient at saturating GPU units, and Contention Inefficiency Loss (CIL), where overlapping tasks fight for resources. By moving communication off the GPU cores, FiCCO significantly lowers CIL. The authors found that DMA-based communication causes far lower slowdowns for the primary computation than traditional GPU-core-driven libraries like RCCL.
Key Results and Insights
The study evaluated FiCCO using real-world GEMM kernels from state-of-the-art LLMs, including Llama-2-70b, Llama-3-405b, Mixtral, and DeepSeek.
- Performance: FiCCO delivers up to a 1.6x speedup over serial execution and significantly outperforms standard shard-level overlap.
- Heuristic Accuracy: The proposed heuristic correctly identifies the optimal execution schedule in 81% of unseen scenarios.
- Contention Reduction: Using DMA engines reduced contention-related slowdowns from ~11% (with standard core-driven communication) to nearly negligible levels in many cases.
- Topology Utilization: On direct-connect meshes, standard P2P patterns can be up to 3.9x slower than serial execution because they leave most links idle. FiCCO’s finer granularity allows for All-to-All patterns that saturate the mesh.
Conclusion
FiCCO demonstrates that moving from coarse shard-level overlap to finer-grained, DMA-offloaded communication can significantly improve the efficiency of distributed ML training. While current hardware limitations exist—such as the lack of native 2D memory copy support in GPU DMA engines—the results suggest that future hardware and compiler integrations (e.g., into Triton or PyTorch) could further unlock the potential of high-bandwidth interconnects.
Summary updated