TACC: A Full-stack Cloud Computing Infrastructure for Machine Learning Tasks

Kaiqiang Xu, Xinchen Wan, Hao Wang, Zhenghang Ren, Xudong Liao, Decang Sun, Chaoliang Zeng, Kai Chen

Abstract

Academic institutions face significant challenges in managing GPU clusters, often caught between the chaos of direct shell access and the complexity of enterprise-grade container platforms. This paper presents SING, a full-stack management system tailored for shared GPU clusters in research environments. SING employs a four-layer architectural abstraction—Job Profile, Adapter, Scheduler, and Execution—to provide a "Machine-Learning-as-a-Service" experience. Evaluation over a 30-month real-world cluster trace demonstrates that SING achieves over 96% GPU utilization while maintaining high reliability and low administrative overhead.

Introduction

The rapid growth of Machine Learning (ML) has led to heavy investment in GPU clusters within university settings. However, managing these resources effectively remains a significant hurdle. Existing solutions often fall into one of three problematic categories:

  1. Direct Shell Access: While simple, it leads to instability as users inadvertently interfere with each other's processes or crash management nodes.
  2. Traditional HPC (e.g., Slurm): Designed for scientific simulations, these systems often struggle with the complex and heterogeneous dependency environments (Conda, Docker, CUDA) typical of modern ML.
  3. Container Platforms (e.g., Kubernetes): While powerful, the engineering overhead required for maintenance is often beyond the capacity of small academic research teams.

SING addresses this "Goldilocks" problem by providing a full-stack solution that balances user-friendly job submission with operational minimalism.

System Architecture

The SING system is organized into four distinct layers, each abstracting a specific part of the cluster management lifecycle.

The four-layer architectural abstraction of the SING system, spanning from job definition to physical execution.

Layer 1: Job Profile

Instead of raw shell commands, users define their tasks via a "Job Profile." This configuration file specifies metadata, scripts, and environment requirements. This approach ensures experiments are repeatable and prevents resource-intensive processes from running on management nodes.

A detailed example of a SING Job Profile, illustrating how users specify resource requirements and software environments.

Layer 2: Adapter Layer

The Adapter layer "compiles" the job profile into an executable format. It automates environment setup by caching Conda environments using MD5 hashes, enabling near-instant startup for recurring jobs. It also provides a Command Line Interface (CLI) for authentication, job submission, and interactive debugging via port forwarding.

Layer 3: Scheduler Layer

SING implements an Optimized First-Come, First-Served (FCFS) scheduling algorithm with backfilling. This strategy prevents large resource requests from blocking the entire queue by filling idle slots with smaller, shorter jobs. To minimize resource fragmentation, the system enforces "Tidy Granularity," allocating GPUs in powers of two.

Layer 4: Execution Layer

The execution layer handles the physical provisioning of resources. While backend-agnostic, the default implementation utilizes a modified Slurm environment. It incorporates SR-IOV for network isolation and GlusterFS for distributed storage, effectively separating temporary job runtime data from persistent user data.

Evaluation and Real-world Impact

The system was evaluated using a comprehensive 30-month trace (July 2021 – December 2023) from a cluster deployed at HKUST. This trace provides fine-grained telemetry on node status and software dependencies, offering deeper insights than previous industry benchmarks.

Key Performance Metrics

  • High Utilization: During peak periods, SING maintained a 96% GPU assignment rate, significantly outperforming standard FCFS implementations (75%).
  • Reliability: The system demonstrated high stability, with only one administrative intervention required for every 940 jobs submitted.
  • Scale: Deployed for over 480 active users, SING managed 160+ GPUs and supported over 28,000 jobs, contributing to approximately 40 peer-reviewed publications.

Discussion and Insights

The deployment of SING revealed several key insights into academic cluster usage:

  • Small Job Incentives: By reserving resources for small-scale jobs (1-2 GPUs), SING encourages users to debug code before requesting large allocations, effectively reducing overall queue wait times.
  • Operational Minimalism: By prioritizing a stable, low-maintenance stack, the authors reduced cluster setup time from 17 to 8 person-days.
  • Human Patterns: Usage peaks between 10 PM and 1 AM and during major paper deadlines highlight the necessity of automated oversubscription management.

Limitations and Future Work

While successful, SING has areas for improvement. Current resource isolation relies on Linux cgroups, which may not prevent all forms of cross-user interference. Future work includes transitioning to a pure HTTPS-based API for enhanced security, implementing self-healing capabilities for node failures, and integrating "bursting" capabilities to public cloud providers.

Summary updated

BibTeX
@inproceedings{xu2021tacc,
  author = {Kaiqiang Xu and Xinchen Wan and Hao Wang and Zhenghang Ren and Xudong Liao and Decang Sun and Chaoliang Zeng and Kai Chen},
  title = {{TACC: A Full-stack Cloud Computing Infrastructure for Machine Learning Tasks}},
  booktitle = {Asplos},
  year = {2021},
}
Figure