Large-scale Machine Learning Cluster Scheduling via Multi-agent Graph Reinforcement Learning

Xiaoyang Zhao, Chuan Wu

Abstract

Efficient scheduling of distributed deep learning (DL) jobs in large GPU clusters is crucial for resource efficiency and job performance. While server sharing among jobs improves resource utilization, interference among co-located DL jobs occurs due to resource contention. Interference-aware job placement has been studied, with white-box approaches based on explicit interference modeling and black-box schedulers with reinforcement learning. In today's clusters containing thousands of GPU servers, running a single scheduler to manage all arrival jobs in a timely and effective manner is challenging, due to the large workload scale. We adopt multiple schedulers in a large-scale cluster/data center, and propose a multi-agent reinforcement learning (MARL) scheduling framework to cooperatively learn fine-grained job placement policies, towards the objective of minimizing job completion time (JCT). To achieve topology-aware placements, our proposed framework uses hierarchical graph neural networks to encode the data center topology and server architecture. In view of a common lack of precise reward samples corresponding to different placements, a job interference model is further devised to predict interference levels in face of various co-locations, for training of the MARL schedulers. Testbed and trace-driven evaluations show that our scheduler framework outperforms representative scheduling schemes by more than 20% in terms of average JCT, and is adaptive to various machine learning cluster topologies.

Introduction

In modern data centers, thousands of GPUs are shared among various deep learning (DL) tasks. Traditional schedulers usually focus on "bin-packing"—trying to fit as many tasks as possible onto the fewest servers to maximize resource utilization. However, this approach often ignores inter-job interference. Even when jobs utilize distinct GPUs, they frequently compete for shared auxiliary resources such as CPU cores, memory bandwidth, or PCIe bus capacity.

Interference impact on various DL models (a) and performance degradation as the number of co-located jobs increases (b).

As shown in Figure 1, this contention leads to significant performance slowdowns, often exceeding 30% for common models like ResNet-50 and VGG-16. Furthermore, as clusters scale to thousands of nodes, centralized scheduling becomes a bottleneck, unable to process job arrivals with the necessary speed or handle the massive state space required for effective reinforcement learning (RL).

Methodology

To address these challenges, we propose a hierarchical, multi-agent framework designed for scalability and interference awareness:

  1. Multi-Agent Reinforcement Learning (MARL): The cluster is partitioned into smaller segments, each managed by an autonomous RL agent. These agents communicate and cooperate to make localized, low-latency placement decisions that optimize for global cluster performance.
  2. Hierarchical Graph Neural Networks (GNNs): To provide agents with a structural understanding of the cluster, we employ GNNs at two levels. An Inner Graph encodes the internal architecture of individual servers (CPUs, GPUs, and PCIe switches), while an Inter-Scheduler Graph captures the network topology (switches and links) between partitions.
  3. Interference Modeling: To facilitate offline training without exhaustive physical profiling, we developed a predictive model that estimates job slowdowns based on CPU utilization and PCIe bus contention. This model provides the necessary reward signals for training the MARL agents.
  4. Actor-Critic Training: Agents are trained using an actor-critic algorithm, with rewards tied to job progress, specifically aiming to minimize the average Job Completion Time (JCT).

Key Insights: The Importance of Topology Awareness

A critical finding of our research is that the specific physical placement within a server—the "socket"—matters significantly. Placing two jobs on the same server but on different CPU sockets (each with its own dedicated PCIe bus) drastically reduces interference compared to sharing a single socket.

The hardware architecture of a typical multi-GPU server (left) and the resulting performance impact of sharing a CPU socket versus using separate sockets (right).

By encoding these architectural nuances into our GNN-based state representation (Figure 2), the scheduler naturally learns to avoid high-contention placements. Additionally, the graph-based approach allows the system to remain topology-aware across different network layouts (e.g., Fat-tree, VL2), placing communication-intensive distributed training tasks on nodes with high-bandwidth, low-latency connections.

Evaluation and Results

We evaluated our framework using realistic workload traces from Google clusters and representative DL models (e.g., ResNet-50, VGG-16) implemented in MXNet.

  • Scalability and Performance: In simulations of 2,000 servers, our framework outperformed industry-standard baselines like Tetris and Load Balancing by over 20% in average JCT.
  • Faster Convergence: The decentralized MARL approach reached optimal policies in approximately 200 epochs, whereas a centralized single-agent baseline failed to converge even after 400 epochs.
  • Real-world Deployment: A prototype on a 48-GPU Kubernetes cluster (AWS EKS) demonstrated an 18.9% improvement over the default Kubernetes scheduler, confirming the practical utility of our approach.

Conclusion

By combining multi-agent cooperation with hierarchical graph-based representations of cluster topology, our framework effectively manages the complexities of large-scale GPU scheduling. It successfully mitigates inter-job interference and scales to thousands of nodes, providing a significant performance boost for modern deep learning workloads.

Summary updated

Abstract

Efficient scheduling of distributed deep learning (DL) jobs in large GPU clusters is crucial for resource efficiency and job performance. While server sharing among jobs improves resource utilization, interference among co-located DL jobs occurs due to resource contention. Interference-aware job placement has been studied, with white-box approaches based on explicit interference modeling and black-box schedulers with reinforcement learning. In today's clusters containing thousands of GPU servers, running a single scheduler to manage all arrival jobs in a timely and effective manner is challenging, due to the large workload scale. We adopt multiple schedulers in a large-scale cluster/data center, and propose a multi-agent reinforcement learning (MARL) scheduling framework to cooperatively learn fine-grained job placement policies, towards the objective of minimizing job completion time (JCT). To achieve topology-aware placements, our proposed framework uses hierarchical graph neural networks to encode the data center topology and server architecture. In view of a common lack of precise reward samples corresponding to different placements, a job interference model is further devised to predict interference levels in face of various co-locations, for training of the MARL schedulers. Testbed and trace-driven evaluations show that our scheduler framework outperforms representative scheduling schemes by more than 20% in terms of average JCT, and is adaptive to various machine learning cluster topologies.
BibTeX
@article{zhao2022large,
  author = {Xiaoyang Zhao and Chuan Wu},
  title = {{Large-scale Machine Learning Cluster Scheduling via Multi-agent Graph Reinforcement Learning}},
  journal = {IEEE Trans. Netw. Serv. Manag.},
  year = {2022},
  url = {https://openreview.net/forum?id=K4axphitRS},
}
Figure