SDCC: software-defined collective communication for distributed training
Lists
Only you see this
Abstract
Communication is crucial to the performance of distributed training. Today’s solutions tightly couple the control and data planes and lack flexibility, generality, and performance. In this study, we present SDCC, a software-defined collective communication framework for distributed training. SDCC is based on the principle of modern systems design to effectively decouple the control plane from the data plane. SDCC abstracts the operations for collective communication in distributed training with dataflow operations and unifies computing and communication with a single dataflow graph. The abstraction, together with the unification, is powerful: it enables users to easily express new and existing collective communication algorithms and optimizations, simplifies the integration with different computing engines (e.g., PyTorch and TensorFlow) and network transports (e.g., Linux TCP and kernel bypass), and allows the system to improve performance by exploiting parallelism exposed by the dataflow graph. We further demonstrate the benefits of SDCC in four use cases.
BibTeX
@article{jin2024sdcc,
author = {Xin Jin and Zhen Zhang and Yunshan Jia and Yun Ma and Xuanzhe Liu},
title = {{SDCC: software-defined collective communication for distributed training}},
journal = {Science China Information Sciences},
year = {2024},
doi = {10.1007/s11432-023-3894-4},
}Loading related content...