Abstract
Algorithms and frameworks for distributed machine learning (DML) have become essential in artificial intelligence engineering. While cloud platforms like Kubernetes provide scalable resource support, native scheduling often fails to address the specific needs of computationally intensive DML jobs, leading to resource deadlocks and inefficiencies. This paper proposes a Combined Priority Scheduling (CPS) method based on Kubernetes and Volcano. By integrating user priority, job urgency, task parallelism, and node affinity, the CPS model optimizes resource allocation and execution order. Experimental results demonstrate that this approach improves job completion efficiency, reduces network latency, and effectively prevents resource deadlocks in cloud environments.
Introduction
Distributed machine learning is inherently resource-intensive, requiring the orchestration of Parameter Servers (PS) to manage model weights and Workers to process data. Standard Kubernetes schedulers often struggle with these workloads due to three primary issues:
- Resource Deadlock: Jobs may partially acquire resources and wait indefinitely for the remainder, stalling the cluster.
- Scheduling Inefficiency: Native schedulers do not inherently recognize the "all-or-nothing" requirement of DML task groups or the high-frequency communication needs between specific nodes.
- Coarse Priority Models: There is often a lack of balance between user importance, job urgency, and technical constraints like task parallelism.
The Combined Priority Scheduling (CPS) Method
The proposed CPS method is built upon Volcano, a CNCF project for high-performance workload scheduling. It utilizes a multi-factor mathematical model to calculate a "Combined Priority" ($V_i$) for each job, mapping these to inter- and intra-group pod scheduling strategies.
Inter-Job and Intra-Job Priority
The scheduling logic is divided into two layers:
- Job Layer (Inter-Job Priority): This determines the execution order between multiple jobs. It factors in User Priority (defined via namespaces), Job Priority (High/Medium/Low), and Time-Constraint Priority (based on maximum tolerable wait time).
- Task Layer (Intra-Job Priority): Within a single job, the system considers Task Parallelism—the number of worker nodes required. By normalizing these disparate metrics into a 0-1 scale, the algorithm ensures that urgent, high-parallelism jobs from high-priority users are prioritized without one factor overwhelming the others.
Affinity and Gang Scheduling
To further optimize performance, the CPS method employs:
- Affinity and Anti-Affinity: The scheduler strategically places Parameter Servers and Workers on the same physical server (affinity) to minimize network latency. Conversely, it uses anti-affinity to prevent multiple Parameter Servers from bottlenecking a single node.
- Gang Scheduling: This "all-or-nothing" strategy ensures a job only begins if the cluster can provide the minimum resources required for the entire pod group, effectively eliminating resource deadlocks.
- PS-First Strategy: Parameter Server pods are given unconditional priority within a job's queue. Since workers depend on the PS, initializing the "brain" of the operation first reduces idle CPU cycles.
Experimental Evaluation
The researchers evaluated the CPS method using the MNIST dataset across two major DML frameworks: TensorFlow and MindSpore. Tests were conducted on clusters of varying sizes (3, 6, and 10 nodes) to ensure scalability.
Key Results
- Performance Gains: The average job implementation time was reduced by over 6% compared to standard Volcano and Kubernetes schedulers.
- Scheduling Accuracy: The success rate for correctly ordering jobs based on the complex priority model exceeded 80%.
- Latency and Deadlock Mitigation: Affinity settings noticeably reduced inter-node communication time, and the gang scheduling strategy successfully prevented cluster deadlocks.
Conclusion and Future Work
The CPS method provides a robust framework for optimizing DML on Kubernetes. However, as AI pipelines evolve into more complex, graph-like workflows, the model may need to incorporate multi-stage dependency handling. Future research will also focus on explicitly modeling heterogeneous resources like GPUs and developing dynamic priority adjustments that respond to real-time training progress or cluster health.
Summary updated