Analyzing Communication Predictability in LLM Training

Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Zhenghang Ren, Xudong Liao, Han Tian, Bo Ren, Zhizhen Zhong, Guyue Liu, Ying Zhang, Kai ChenarXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) continue to scale, the efficiency of distributed training is increasingly bottlenecked by communication overhead, which can account for up to 55% of total execution time. Current optimization strategies rely on reactive online profiling, which is both computationally expensive and time-consuming. This paper presents a first-principles mathematical framework for predicting communication patterns in LLM training, covering both dense models and dynamic Mixture-of-Experts (MoE) architectures. By formalizing the spatial and temporal dimensions of data movement, we introduce ConfigTuner, an analytical tool that optimizes training throughput without the need for exhaustive trial-and-error. Our evaluation across diverse hardware generations and model scales (up to 145B parameters) demonstrates that communication is highly predictable, allowing ConfigTuner to achieve up to 1.36× throughput gains with ~95% prediction accuracy.

Introduction

Training modern LLMs requires "hybrid parallelism"—a complex orchestration of data, pipeline, and tensor sharding across thousands of GPUs. In these environments, the primary challenge is not just raw computation, but the "logistics" of inter-GPU communication. Historically, engineers have optimized these configurations through online profiling: running a model, observing bottlenecks, and manually adjusting settings. This approach is reactive and fails to provide a fundamental understanding of why certain configurations outperform others.

This research addresses this gap by asking whether communication overhead can be predicted mathematically before training begins. We demonstrate that the "choreography" of clusters is remarkably stable, allowing for a shift from empirical profiling to predictive modeling.

Methodology

1. Analytical Formulation of Communication

We decompose communication into two primary dimensions:

  • Spatial Dimension: Determining who talks to whom and the volume of data exchanged. This is derived from the model's architectural parameters (e.g., hidden layer size, number of heads) and the chosen parallelism strategy.
  • Temporal Dimension: Determining when communication occurs. We model the rigid "On-Off" rhythm of transmission inherent in the synchronous nature of LLM training cycles.

2. Decoding MoE Predictability

Mixture-of-Experts (MoE) models are often considered unpredictable due to dynamic routing, where data is sent to different "expert" neurons based on input content. However, our analysis reveals a "semi-predictability" in MoE traffic. We identified a consistency rule: the four AllToAll communication events within a single MoE layer are either identical or transposes of one another. Furthermore, as training progresses and the gating network matures, traffic patterns stabilize and become increasingly uniform, making them amenable to analytical modeling.

3. ConfigTuner

We integrated these mathematical insights into ConfigTuner. Unlike existing auto-parallelism tools like Alpa, which require executing multiple full iterations to gather statistics, ConfigTuner utilizes minimal micro-benchmarks—such as single-GPU compute speed and local network latency—to predict the performance of any given configuration. This significantly reduces the search complexity for optimal training setups.

Experimental Setup

To validate the framework, we utilized:

  • Models: A suite of GPT-style models (1.5B to 145B parameters) and a 1.3B parameter MoE model.
  • Data: Production traces from five large-scale DNN models provided by industry partners.
  • Hardware: Four generations of NVIDIA architectures, including PCIe-based GeForce, NVLink-based Tesla, and NVSwitch-equipped Hopper systems.

Key Insights

  • The Stability of GPU Utilization (μ): We discovered that GPU utilization remains stable as long as the per-GPU model size and micro-batch size are constant, regardless of total cluster scale. This allows for reliable performance extrapolation.
  • Effective vs. Theoretical Bandwidth: Our model explicitly accounts for the gap between theoretical hardware limits and effective bandwidth, which is often degraded by synchronization overhead and small message sizes.
  • Predictable Choreography: For dense models, the fixed architecture results in a near-perfectly predictable data movement pattern, allowing for traffic volume calculations with high precision.

Results

ConfigTuner demonstrates high fidelity to real-world performance, achieving approximately 95% accuracy in predicting iteration times and communication-to-computation ratios. When compared to the default settings in Megatron-LM, ConfigTuner-suggested configurations improved throughput by up to 1.36×. Notably, it reached the same optimal configurations as Alpa but with a fraction of the search time and zero execution overhead.

Conclusion and Future Work

This work proves that the communication patterns in LLM training are not chaotic but follow a predictable mathematical structure. While our current model assumes a dedicated network environment and follows the standard 1F1B pipeline schedule, it provides a robust foundation for automated training optimization. Future research will extend this framework to multi-tenant datacenter scenarios where network congestion and link competition introduce additional variables into the communication equation.

Summary updated

Abstract

Effective communication is essential in distributed training, with predictability being one of its most significant characteristics. However, existing studies primarily focus on exploiting predictability through online profiling for runtime optimization, without a systematic understanding of it. In this work, we aim to systematically formulate communication predictability in distributed training, particularly in Large Language Models (LLMs) that utilize hybrid parallelism. Our analysis focuses on both traffic patterns and communication overhead. Specifically, we investigate predictable traffic patterns in typical LLMs and evaluate how various factors influence GPU utilization and effective bandwidth (two critical variables affecting communication overhead). Furthermore, we develop an analytical formulation to estimate communication overhead in LLM training, which is validated with high accuracy against empirical data. Leveraging this formulation, we propose a configuration tuning tool, ConfigTuner, to optimize training performance. Compared to Megatron-LM, the training configurations optimized by ConfigTuner demonstrate up to a 1.36$\times$ increase in throughput. Compared to Alpa, ConfigTuner generates the same configuration suggestion while significantly reducing the search complexity.
BibTeX
@misc{li2025analyzing,
  author = {Wenxue Li and Xiangzhou Liu and Yuxuan Li and Yilun Jin and Zhenghang Ren and Xudong Liao and Han Tian and Bo Ren and Zhizhen Zhong and Guyue Liu and Ying Zhang and Kai Chen},
  title = {{Analyzing Communication Predictability in LLM Training}},
  year = {2025},
  eprint = {2512.24750},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.24750},
}
Figure