A combined priority scheduling method for distributed machine learning

Tiantian Du, Gongyi Xiao, Jing Chen, ChuanFu Zhang, Hao Sun, Li Wen, Yudong GengEURASIP 2023 · DOI

Abstract

Algorithms and frameworks for distributed machine learning (DML) have become essential in artificial intelligence engineering. While cloud platforms like Kubernetes provide scalable resource support, native scheduling often fails to address the specific needs of computationally intensive DML jobs, leading to resource deadlocks and inefficiencies. This paper proposes a Combined Priority Scheduling (CPS) method based on Kubernetes and Volcano. By integrating user priority, job urgency, task parallelism, and node affinity, the CPS model optimizes resource allocation and execution order. Experimental results demonstrate that this approach improves job completion efficiency, reduces network latency, and effectively prevents resource deadlocks in cloud environments.

Introduction

Distributed machine learning is inherently resource-intensive, requiring the orchestration of Parameter Servers (PS) to manage model weights and Workers to process data. Standard Kubernetes schedulers often struggle with these workloads due to three primary issues:

  1. Resource Deadlock: Jobs may partially acquire resources and wait indefinitely for the remainder, stalling the cluster.
  2. Scheduling Inefficiency: Native schedulers do not inherently recognize the "all-or-nothing" requirement of DML task groups or the high-frequency communication needs between specific nodes.
  3. Coarse Priority Models: There is often a lack of balance between user importance, job urgency, and technical constraints like task parallelism.

The Combined Priority Scheduling (CPS) Method

The proposed CPS method is built upon Volcano, a CNCF project for high-performance workload scheduling. It utilizes a multi-factor mathematical model to calculate a "Combined Priority" ($V_i$) for each job, mapping these to inter- and intra-group pod scheduling strategies.

Architecture of the CPS model, illustrating the Job and Task layers.

Inter-Job and Intra-Job Priority

The scheduling logic is divided into two layers:

  • Job Layer (Inter-Job Priority): This determines the execution order between multiple jobs. It factors in User Priority (defined via namespaces), Job Priority (High/Medium/Low), and Time-Constraint Priority (based on maximum tolerable wait time).
  • Task Layer (Intra-Job Priority): Within a single job, the system considers Task Parallelism—the number of worker nodes required. By normalizing these disparate metrics into a 0-1 scale, the algorithm ensures that urgent, high-parallelism jobs from high-priority users are prioritized without one factor overwhelming the others.

Affinity and Gang Scheduling

To further optimize performance, the CPS method employs:

  • Affinity and Anti-Affinity: The scheduler strategically places Parameter Servers and Workers on the same physical server (affinity) to minimize network latency. Conversely, it uses anti-affinity to prevent multiple Parameter Servers from bottlenecking a single node.
  • Gang Scheduling: This "all-or-nothing" strategy ensures a job only begins if the cluster can provide the minimum resources required for the entire pod group, effectively eliminating resource deadlocks.
  • PS-First Strategy: Parameter Server pods are given unconditional priority within a job's queue. Since workers depend on the PS, initializing the "brain" of the operation first reduces idle CPU cycles.

Experimental Evaluation

The researchers evaluated the CPS method using the MNIST dataset across two major DML frameworks: TensorFlow and MindSpore. Tests were conducted on clusters of varying sizes (3, 6, and 10 nodes) to ensure scalability.

Key Results

  • Performance Gains: The average job implementation time was reduced by over 6% compared to standard Volcano and Kubernetes schedulers.
  • Scheduling Accuracy: The success rate for correctly ordering jobs based on the complex priority model exceeded 80%.
  • Latency and Deadlock Mitigation: Affinity settings noticeably reduced inter-node communication time, and the gang scheduling strategy successfully prevented cluster deadlocks.

Conclusion and Future Work

The CPS method provides a robust framework for optimizing DML on Kubernetes. However, as AI pipelines evolve into more complex, graph-like workflows, the model may need to incorporate multi-stage dependency handling. Future research will also focus on explicitly modeling heterogeneous resources like GPUs and developing dynamic priority adjustments that respond to real-time training progress or cluster health.

Summary updated

Abstract

Abstract Algorithms and frameworks for distributed machine learning have been widely used in numerous artificial intelligence engineering applications. A cloud platform provides a large number of resources at a lower cost and is a more convenient method for such applications. With the rapid development of containerization, native cloud combinations based on Docker and Kubernetes have provided effective resource support for distributed machine learning. However, native Kubernetes does not provide efficient priority or fair resource scheduling strategies for distributed machine learning in computationally intensive and time-consuming jobs, which easily leads to resource deadlock, resource waste, and low job execution efficiency. Therefore, to utilize the execution order between multiple jobs in distributed machine learning as well as the dependencies between multiple tasks for the same job, considering intra- and inter-group scheduling priorities, a combined priority scheduling method is proposed for distributed machine learning based on Kubernetes and Volcano. Considering the user priority, task priority, longest wait time, task parallelism, and affinity and non-affinity between the parameter server and worker nodes, a combined priority scheduling model of inter- and intra-job priority is proposed, which is mapped into a scheduling strategy of inter- and intra-group priorities of pods, enabling the efficient scheduling and training of distributed machine learning. The experiment results show that the proposed method achieves preferential resource allocation for urgent, high parallelism, and high-priority jobs with high-priority users and improves the job execution efficiency. The affinity and anti-affinity settings among pods reduce the time of information interaction between the parameter server and worker nodes to a certain extent, thereby improving the job completion efficiency. This group scheduling strategy alleviates the problems of resource deadlock and waste caused by insufficient resources in cloud computing.
BibTeX
@inproceedings{du2023combined,
  author = {Tiantian Du and Gongyi Xiao and Jing Chen and ChuanFu Zhang and Hao Sun and Li Wen and Yudong Geng},
  title = {{A combined priority scheduling method for distributed machine learning}},
  booktitle = {EURASIP},
  year = {2023},
  doi = {10.1186/s13638-023-02253-4},
}
Figure