Optimal Distributed Training With Co-Adaptive Data Parallelism in Heterogeneous Environments

Lifang Chen, Zhichao Chen, Liqi Yan, Yanyu Cheng, Fangli Guan, Pan LiIJCAI 2025 · DOI

TL;DR

This paper introduces Co-Adaptive Data Parallelism (C-ADP), a novel distributed training framework designed to tackle the challenges of training large deep learning models across heterogeneous computing environments. C-ADP optimizes data distribution and employs delayed synchronization to significantly reduce training time and improve efficiency compared to existing methods.

What Problem Does This Paper Address?

The demand for training increasingly massive deep learning models, from neural networks to Large Language Models (LLMs), has exploded due to the exponential growth of data. This necessitates distributed training to leverage distributed computational power. However, existing frameworks like DDP and DeepSpeed are primarily designed for homogeneous, co-located clusters. They struggle with geo-distributed environments characterized by varying computing capabilities (e.g., GPUs vs. CPUs) and communication bandwidths. This heterogeneity leads to inefficient data distribution, uneven workloads, and communication bottlenecks, hindering optimal training performance.

Methods

C-ADP addresses these challenges through a three-pronged approach:

  • Local Training and Delayed Parameter Synchronization: Instead of synchronizing parameters after every batch, C-ADP synchronizes them at the end of each epoch. This significantly reduces communication overhead, especially in environments with limited bandwidth.
  • Data Parallel Scheduling Optimization: The core innovation is formulating data distribution as an optimization problem. The goal is to minimize the total training time by intelligently allocating data samples to each computing device. This involves balancing data allocation time, computation time, and communication time.
  • Efficient and Optimal Algorithm: The paper proposes an efficient algorithm to solve this optimization problem. They formally prove that the obtained solution is optimal in the asymptotic sense, meaning it approaches the true optimum as the number of data samples becomes very large. This algorithm ensures balanced loading across devices with diverse capabilities.

Datasets

  • ImageNet100: Used for training and evaluation. This is a subset of the ImageNet dataset, consisting of 100 classes, 100,000 training samples, and 10,000 test samples.

Key Insights and Ideas

  • Heterogeneity is the New Normal: The paper highlights that real-world distributed training often involves diverse hardware (GPUs, CPUs) and network conditions, and existing frameworks are not well-equipped for this.
  • Data Distribution is Key: The most impactful insight is that intelligently distributing data, rather than using uniform distribution, is crucial for optimizing training time in heterogeneous settings. C-ADP's scheduler dynamically determines the optimal data allocation.
  • Delayed Synchronization is a Trade-off Worth Making: While delayed synchronization might seem to slow down convergence initially, the significant reduction in communication overhead can lead to faster overall training times, especially in bandwidth-constrained environments.
  • Asymptotic Optimality: The theoretical guarantee of asymptotic optimality for their scheduling algorithm provides strong confidence in its effectiveness for large-scale training scenarios.

Key Results

  • Significant Speedup: C-ADP achieved 21.6 times improvement in FLOPS over DDP and 26.3 times improvement over DeepSpeed.
  • Reduced Training Time: C-ADP reduced training time by approximately 72% compared to DDP and 47% compared to DeepSpeed.
  • Improved Accuracy: On the ImageNet100 dataset, C-ADP achieved 73.67% accuracy at the 50th epoch, surpassing single-card training and other distributed methods.
  • Efficient Data Transfer: C-ADP required only 431.82s for data transfer, significantly less than DDP (1411.68s) and DeepSpeed (1398.46s).
  • Balanced Loading: The framework demonstrated even loading across heterogeneous devices, addressing a key challenge in distributed training.

Limitations and Future Work

  • Asymptotic Optimality: While the algorithm is proven optimal in the asymptotic sense, its performance for smaller datasets or very limited numbers of devices might differ.
  • Focus on Data Parallelism: The current framework focuses solely on data parallelism. Future work could explore integrating C-ADP with other parallelism strategies (e.g., pipeline or tensor parallelism) for even greater efficiency.
  • Real-world Network Emulation: The paper mentions simulating network heterogeneity. Further validation with actual, dynamic network conditions could provide deeper insights.
  • Hyperparameter Tuning: While the paper details hyperparameter configurations, exploring automated hyperparameter optimization for C-ADP in heterogeneous environments could be a valuable future direction.

Summary updated

BibTeX
@inproceedings{chen2025optimal,
  author = {Lifang Chen and Zhichao Chen and Liqi Yan and Yanyu Cheng and Fangli Guan and Pan Li},
  title = {{Optimal Distributed Training With Co-Adaptive Data Parallelism in Heterogeneous Environments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/4},
}
Figure