Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

Andrew Sheinberg, Samyak Gupta, Kai LiarXiv 2026 · arXiv

TL;DR

Researchers at Princeton University deployed a fine-grained monitoring tool to analyze a production GPU cluster at 100ms intervals, revealing that these expensive resources are drastically underutilized. By capturing sub-second activity, the study identifies a "middle layer" of opportunity—better job scheduling and GPU multitasking—that could recover massive amounts of wasted computational power without requiring new hardware or application rewrites.

What Problem Does This Paper Address?

GPU clusters are the backbone of modern AI, but they are incredibly expensive and often plagued by long wait times. The standard way we manage these clusters is inefficient: we typically allocate an entire GPU to a single job for its entire duration, even if that job only uses a fraction of the hardware's capabilities.

Furthermore, existing monitoring tools like nvidia-smi often operate at second- or minute-level intervals. This "coarse-grained" view misses the rapid bursts of activity and long idle gaps inherent in many workloads. The authors argue that this lack of visibility leads to poor scheduling decisions and leaves significant "room in the middle"—unused capacity that could be reclaimed if we managed resources more intelligently.

Methods

The authors developed a lightweight, open-source monitoring tool written in C that leverages NVIDIA’s Data Center GPU Manager (DCGM) library.

  • Fine-Grained Sampling: Unlike standard tools, their solution captures metrics every 100ms. This aligns with the default Linux round-robin time slice, allowing them to see "inside" the typical scheduling window.
  • Precise Metrics: They focused on "SM Activity" (Streaming Multiprocessor utilization) rather than just kernel occupancy. This distinguishes between a job that uses all 108 SMs on an A100 and one that uses just one, both of which might be reported as "100% utilized" by coarser tools.
  • Low Overhead: The tool was designed for production environments, consuming less than 1% of a single CPU core and negligible GPU resources.
Comparison of nvidia-smi (NVML) vs. DCGM reporting. Standard metrics significantly overestimate actual SM usage.

Datasets

The study analyzed a university research cluster consisting of 89 nodes and 319 NVIDIA A100 GPUs (both PCIe 40GB and SXM4 80GB versions). The primary analysis focuses on a 10-day period featuring:

  • 7,399 submitted jobs.
  • 150 unique users.
  • 12 different academic departments (ranging from Chemistry to Sociology).

Key Insights and Ideas

  • The nvidia-smi Illusion: Standard monitoring tools are misleading. A job using only a single SM can appear to have 100% utilization because a kernel is technically "active." Fine-grained monitoring reveals that many "busy" GPUs are actually mostly idle at the hardware level.
  • The Duration Guessing Game: Users are remarkably poor at estimating how long their jobs will take. Approximately 80% of jobs showed a drastic mismatch between requested and actual runtimes. This inaccuracy wreaks havoc on cluster schedulers, leading to inflated queue times and fragmented resources.
  • Predictable Rhythms: Multi-GPU jobs exhibit highly periodic communication patterns (via NVLink and InfiniBand). These "heartbeats" of activity suggest that schedulers could potentially interleave other tasks during the predictable idle phases of a large distributed job.
Fig. 2. CDF of job elapsed time (𝑁= 7, 399). Colored segments correspond to our different duration categories.

Key Results

  • Severe Underutilization: Over 70% of jobs used less than 20% of their allocated GPU memory. Even more striking, 50% of jobs averaged below 5% SM utilization.
  • Queue Bottlenecks: While 95% of jobs only requested a single GPU, the few multi-GPU and multi-node jobs consumed a disproportionate amount of cluster time and faced median queue delays of up to 2 days, compared to 2 hours for single-GPU requests.
  • Wasted Time: Roughly 34% of jobs timed out, meaning they were killed before completion, wasting all the GPU cycles they had consumed. Conversely, many jobs were "fully idle," reserving a GPU for hours without ever launching a kernel.
Queue times increase significantly for multi-GPU and multi-node requests.

Limitations and Future Work

The authors acknowledge that while 100ms is much better than 1-second sampling, individual GPU kernels often run on microsecond timescales. Even finer granularity might reveal even more "burstiness," though it would increase monitoring overhead.

The study was also limited to a 10-day deep dive on a specific university cluster. Future work should explore how these patterns hold up in large-scale industrial AI labs. The most promising path forward is the development of dynamic, transparent multitasking—systems that can automatically pack multiple under-utilized jobs onto the same GPU using technologies like NVIDIA's Multi-Process Service (MPS) or Multi-Instance GPU (MIG), guided by the fine-grained telemetry introduced here.

Summary updated

Abstract

GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While previous monitoring efforts have operated at second- or minute-level frequencies, they primarily informed job-level scheduling for clusters and lacked the detail needed to optimize GPU-level utilization. This paper presents an in-depth study of a university research GPU cluster, analyzing resource usage at 100 ms intervals. We have developed a lightweight monitoring tool that leverages Nvidia's DCGM library to capture fine-grained "compute utilization" metrics within GPUs. Deployed on a cluster with 89 nodes and 319 GPUs, our tool collected data over several months. This paper focuses on a 10-day period with 7,399 jobs submitted by 150 users from 12 departments. Our study reveals three key findings: (1) the GPU cluster exhibits significant underutilization, with over 70% of jobs using less than 20% of GPU memory and 50% averaging below 5% Streaming Multiprocessor (SM) utilization; (2) 80% of jobs have a drastic mismatch between their requested and actual durations, significantly limiting the scheduler's effectiveness; and (3) the multi-GPU jobs we examined show highly periodic communication patterns. These findings suggest there is plenty of room in the middle: substantial utilization gains are available through better job scheduling and GPU multitasking, independent of hardware or application improvements. Our monitoring solution is available as open-source software at https://github.com/als244/gpu_server_monitoring.
BibTeX
@misc{sheinberg2026plenty,
  author = {Andrew Sheinberg and Samyak Gupta and Kai Li},
  title = {{Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring}},
  year = {2026},
  eprint = {2610.36329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36329},
}
Figure