Abstract
Academic institutions face significant challenges in managing GPU clusters, often caught between the chaos of direct shell access and the complexity of enterprise-grade container platforms. This paper presents SING, a full-stack management system tailored for shared GPU clusters in research environments. SING employs a four-layer architectural abstraction—Job Profile, Adapter, Scheduler, and Execution—to provide a "Machine-Learning-as-a-Service" experience. Evaluation over a 30-month real-world cluster trace demonstrates that SING achieves over 96% GPU utilization while maintaining high reliability and low administrative overhead.
Introduction
The rapid growth of Machine Learning (ML) has led to heavy investment in GPU clusters within university settings. However, managing these resources effectively remains a significant hurdle. Existing solutions often fall into one of three problematic categories:
- Direct Shell Access: While simple, it leads to instability as users inadvertently interfere with each other's processes or crash management nodes.
- Traditional HPC (e.g., Slurm): Designed for scientific simulations, these systems often struggle with the complex and heterogeneous dependency environments (Conda, Docker, CUDA) typical of modern ML.
- Container Platforms (e.g., Kubernetes): While powerful, the engineering overhead required for maintenance is often beyond the capacity of small academic research teams.
SING addresses this "Goldilocks" problem by providing a full-stack solution that balances user-friendly job submission with operational minimalism.
System Architecture
The SING system is organized into four distinct layers, each abstracting a specific part of the cluster management lifecycle.
Layer 1: Job Profile
Instead of raw shell commands, users define their tasks via a "Job Profile." This configuration file specifies metadata, scripts, and environment requirements. This approach ensures experiments are repeatable and prevents resource-intensive processes from running on management nodes.
Layer 2: Adapter Layer
The Adapter layer "compiles" the job profile into an executable format. It automates environment setup by caching Conda environments using MD5 hashes, enabling near-instant startup for recurring jobs. It also provides a Command Line Interface (CLI) for authentication, job submission, and interactive debugging via port forwarding.
Layer 3: Scheduler Layer
SING implements an Optimized First-Come, First-Served (FCFS) scheduling algorithm with backfilling. This strategy prevents large resource requests from blocking the entire queue by filling idle slots with smaller, shorter jobs. To minimize resource fragmentation, the system enforces "Tidy Granularity," allocating GPUs in powers of two.
Layer 4: Execution Layer
The execution layer handles the physical provisioning of resources. While backend-agnostic, the default implementation utilizes a modified Slurm environment. It incorporates SR-IOV for network isolation and GlusterFS for distributed storage, effectively separating temporary job runtime data from persistent user data.
Evaluation and Real-world Impact
The system was evaluated using a comprehensive 30-month trace (July 2021 – December 2023) from a cluster deployed at HKUST. This trace provides fine-grained telemetry on node status and software dependencies, offering deeper insights than previous industry benchmarks.
Key Performance Metrics
- High Utilization: During peak periods, SING maintained a 96% GPU assignment rate, significantly outperforming standard FCFS implementations (75%).
- Reliability: The system demonstrated high stability, with only one administrative intervention required for every 940 jobs submitted.
- Scale: Deployed for over 480 active users, SING managed 160+ GPUs and supported over 28,000 jobs, contributing to approximately 40 peer-reviewed publications.
Discussion and Insights
The deployment of SING revealed several key insights into academic cluster usage:
- Small Job Incentives: By reserving resources for small-scale jobs (1-2 GPUs), SING encourages users to debug code before requesting large allocations, effectively reducing overall queue wait times.
- Operational Minimalism: By prioritizing a stable, low-maintenance stack, the authors reduced cluster setup time from 17 to 8 person-days.
- Human Patterns: Usage peaks between 10 PM and 1 AM and during major paper deadlines highlight the necessity of automated oversubscription management.
Limitations and Future Work
While successful, SING has areas for improvement. Current resource isolation relies on Linux cgroups, which may not prevent all forms of cross-user interference. Future work includes transitioning to a pure HTTPS-based API for enhanced security, implementing self-healing capabilities for node failures, and integrating "bursting" capabilities to public cloud providers.
Summary updated