TL;DR
Researchers at Princeton University deployed a fine-grained monitoring tool to analyze a production GPU cluster at 100ms intervals, revealing that these expensive resources are drastically underutilized. By capturing sub-second activity, the study identifies a "middle layer" of opportunity—better job scheduling and GPU multitasking—that could recover massive amounts of wasted computational power without requiring new hardware or application rewrites.
What Problem Does This Paper Address?
GPU clusters are the backbone of modern AI, but they are incredibly expensive and often plagued by long wait times. The standard way we manage these clusters is inefficient: we typically allocate an entire GPU to a single job for its entire duration, even if that job only uses a fraction of the hardware's capabilities.
Furthermore, existing monitoring tools like nvidia-smi often operate at second- or minute-level intervals. This "coarse-grained" view misses the rapid bursts of activity and long idle gaps inherent in many workloads. The authors argue that this lack of visibility leads to poor scheduling decisions and leaves significant "room in the middle"—unused capacity that could be reclaimed if we managed resources more intelligently.
Methods
The authors developed a lightweight, open-source monitoring tool written in C that leverages NVIDIA’s Data Center GPU Manager (DCGM) library.
- Fine-Grained Sampling: Unlike standard tools, their solution captures metrics every 100ms. This aligns with the default Linux round-robin time slice, allowing them to see "inside" the typical scheduling window.
- Precise Metrics: They focused on "SM Activity" (Streaming Multiprocessor utilization) rather than just kernel occupancy. This distinguishes between a job that uses all 108 SMs on an A100 and one that uses just one, both of which might be reported as "100% utilized" by coarser tools.
- Low Overhead: The tool was designed for production environments, consuming less than 1% of a single CPU core and negligible GPU resources.
Datasets
The study analyzed a university research cluster consisting of 89 nodes and 319 NVIDIA A100 GPUs (both PCIe 40GB and SXM4 80GB versions). The primary analysis focuses on a 10-day period featuring:
- 7,399 submitted jobs.
- 150 unique users.
- 12 different academic departments (ranging from Chemistry to Sociology).
Key Insights and Ideas
- The
nvidia-smiIllusion: Standard monitoring tools are misleading. A job using only a single SM can appear to have 100% utilization because a kernel is technically "active." Fine-grained monitoring reveals that many "busy" GPUs are actually mostly idle at the hardware level. - The Duration Guessing Game: Users are remarkably poor at estimating how long their jobs will take. Approximately 80% of jobs showed a drastic mismatch between requested and actual runtimes. This inaccuracy wreaks havoc on cluster schedulers, leading to inflated queue times and fragmented resources.
- Predictable Rhythms: Multi-GPU jobs exhibit highly periodic communication patterns (via NVLink and InfiniBand). These "heartbeats" of activity suggest that schedulers could potentially interleave other tasks during the predictable idle phases of a large distributed job.
Key Results
- Severe Underutilization: Over 70% of jobs used less than 20% of their allocated GPU memory. Even more striking, 50% of jobs averaged below 5% SM utilization.
- Queue Bottlenecks: While 95% of jobs only requested a single GPU, the few multi-GPU and multi-node jobs consumed a disproportionate amount of cluster time and faced median queue delays of up to 2 days, compared to 2 hours for single-GPU requests.
- Wasted Time: Roughly 34% of jobs timed out, meaning they were killed before completion, wasting all the GPU cycles they had consumed. Conversely, many jobs were "fully idle," reserving a GPU for hours without ever launching a kernel.
Limitations and Future Work
The authors acknowledge that while 100ms is much better than 1-second sampling, individual GPU kernels often run on microsecond timescales. Even finer granularity might reveal even more "burstiness," though it would increase monitoring overhead.
The study was also limited to a 10-day deep dive on a specific university cluster. Future work should explore how these patterns hold up in large-scale industrial AI labs. The most promising path forward is the development of dynamic, transparent multitasking—systems that can automatically pack multiple under-utilized jobs onto the same GPU using technologies like NVIDIA's Multi-Process Service (MPS) or Multi-Instance GPU (MIG), guided by the fine-grained telemetry introduced here.
Summary updated