CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training

Yida Gu, Fakang Wang, Jianhao Fu, Zhenhang Sun, Qianyu Zhang, Hairui Zhao, Xingchen Liu, Yang Tian, Wenjing Huang, Zedong Liu, Yifan Chen, Jinwu Yang, Yueyuan Zhou, Qian Zhao, Haoxu Li, Tao Wang, Feng Yu, Zhan Wang, Guangming Tan, Dingwen Tao

TL;DR

Large-scale model training is plagued by "slow/hang" communication anomalies that are notoriously hard to diagnose, leading to significant downtime. This paper introduces CCL-D, a novel system that precisely detects and locates these anomalies within minutes by analyzing fine-grained, cross-layer communication metrics, drastically outperforming existing methods.

What Problem Does This Paper Address?

The relentless scaling of deep learning models, particularly large language models (LLMs), has pushed distributed training to unprecedented scales. While this scaling unlocks greater capabilities, it also exacerbates system reliability issues. Collective communication libraries (CCLs), which are fundamental for synchronizing model parameters across thousands of GPUs, are increasingly prone to anomalies. Among these, "slow/hang" communication anomalies are the most frequent and time-consuming to diagnose. Traditional methods are often inaccurate, inefficient, and require hours or even days for root cause analysis, making them impractical for timely mitigation. This gap in diagnostic capabilities hinders the advancement of large-scale distributed training.

Methods

CCL-D tackles this problem with a two-pronged approach:

  • Rank Probing Module: This module is deployed on each participating rank (GPU) and collects a carefully designed set of multi-level metrics. These metrics span both host and GPU kernel states, focusing on atomic Send/Recv operations. This cross-layer approach provides fine-grained visibility into communication traffic, aiming to pinpoint anomalies without relying on coarse-grained host-level information. The design prioritizes minimizing the number of metrics to maximize diagnostic coverage and utility while keeping probing complexity low.
  • Decision Analysis Module: This centralized, yet scalable, component analyzes the metrics collected by the probing module. It performs automated anomaly detection and then employs specialized decision algorithms to accurately locate the root-cause ranks. This module operates out-of-band, meaning it doesn't interfere with the training execution itself. The analysis is designed to be efficient, aiming to diagnose anomalies within minutes.

Key novelties include:

  • Cross-layer metrics: CCL-D leverages metrics from both host and kernel layers, specifically focusing on Send/Recv counts and rates, which are fundamental to communication.
  • Metric-Aware Tracing Framework: A lightweight distributed tracing framework is implemented to capture communication traffic with minimal overhead (<1%). This framework uses a "Trace ID" for decentralized labeling of communication rounds and a "Probing Frame" for efficient kernel metric recording.
  • Host-Driven Measurement: To avoid impacting GPU performance, most metric measurement and computation are shifted to the CPU, utilizing CUDA Unified Virtual Addressing (UVA) for efficient memory sharing and implicit DMA propagation.

Datasets

The paper does not explicitly mention specific datasets used for training the models evaluated. However, it lists several large language models used for training and evaluation:

  • BaiLing-5B
  • Llama2-7B
  • Llama3.1-8B
  • BaiLing-80B

These models were trained using two strategies: Fully Sharded Data Parallelism (FSDP) and 3D parallelism. The evaluation also involved datasets like Alpaca and Fineweb-edu for training the models.

Key Insights and Ideas

  • The critical role of Send/Recv primitives: The authors argue that atomic Send and Recv operations are fundamental primitives for collective communication. Their behavior, independent of underlying hardware or protocols, makes them ideal for identifying anomalies.
  • Fine-grained metrics are key: Simply monitoring operation counts or host-level information is insufficient. Analyzing kernel-level Send/Recv counts and rates provides the necessary precision to distinguish between different types of slow/hang anomalies.
  • Host-driven measurement is crucial for low overhead: Shifting metric collection and analysis to the host CPU, rather than directly on the GPU, is essential to avoid impacting training performance and to achieve scalability.
  • Decoupling diagnosis from training: Running the diagnostic system out-of-band ensures that it doesn't introduce performance bottlenecks or affect the accuracy of the training process.
  • Scalability is paramount: The proposed system is designed to scale to thousands of GPUs, a critical requirement for modern large-scale training.

Key Results

  • High Accuracy and Coverage: CCL-D achieves comprehensive coverage of both slow/hang anomalies and precise rank-level attribution, outperforming baseline methods like Bisection, Stack Analysis, RAS, Greyhound, and C4D.
  • Exceptional Efficiency: CCL-D diagnoses slow/hang anomalies with high precision within 6 minutes on a 4,000-GPU cluster, a dramatic improvement over existing methods that can take hours or days.
  • Minimal Overhead: The tracing framework introduces negligible overhead (<1%) during normal training. CPU overhead is consistently low (around 0.3% per node) even at large scales. Communication operation measurements introduce minimal additional cost (<0.45%).
  • Scalability Demonstrated: The system scales effectively to 4,000 GPUs, maintaining its diagnostic speed and accuracy.
  • No Impact on Training Accuracy: Experiments show that using CCL-D does not negatively impact the training loss function, indicating no adverse effect on model convergence.

Limitations and Future Work

  • Limitations:
    • While CCL-D achieves high precision, the paper acknowledges that "mixed-slow" scenarios can still be challenging to attribute to a single root cause.
  • The paper focuses on CCL-level anomalies and does not directly address broader system-level issues outside of communication.
  • Future Work:
    • The authors suggest that combining CCL-D with other diagnostic tools (e.g., network/system-level monitoring) could offer even more comprehensive coverage for large-scale training.
  • Further refinement of anomaly classification and root-cause attribution for more complex scenarios could be explored.

Summary updated

Abstract

As training scales grow, collective communication libraries (CCL) increasingly face anomalies arising from complex interactions among hardware, software, and environmental factors. These anomalies typically manifest as slow/hang communication, the most frequent and time-consuming category to diagnose. However, traditional diagnostic methods remain inaccurate and inefficient, frequently requiring hours or even days for root cause analysis. To address this, we propose CCL-D, a high-precision diagnostic system designed to detect and locate slow/hang anomalies in large-scale distributed training. CCL-D integrates a rank-level real-time probe with an intelligent decision analyzer. The probe measures cross-layer anomaly metrics using a lightweight distributed tracing framework to monitor communication traffic. The analyzer performs automated anomaly detection and root-cause location, precisely identifying the faulty GPU rank. Deployed on a 4,000-GPU cluster over one year, CCL-D achieved near-complete coverage of known slow/hang anomalies and pinpointed affected ranks within 6 minutes-substantially outperforming existing solutions.
BibTeX
@inproceedings{gu2026ccl,
  author = {Yida Gu and Fakang Wang and Jianhao Fu and Zhenhang Sun and Qianyu Zhang and Hairui Zhao and Xingchen Liu and Yang Tian and Wenjing Huang and Zedong Liu and Yifan Chen and Jinwu Yang and Yueyuan Zhou and Qian Zhao and Haoxu Li and Tao Wang and Feng Yu and Zhan Wang and Guangming Tan and Dingwen Tao},
  title = {{CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training}},
  booktitle = {PPoPP},
  year = {2026},
  doi = {10.1145/3774934.3786429},
}

Loading related content...

Figure