Co-locating Workloads on Shared Hardware Accelerators

30 concepts21 papers1 proposal

Hardware accelerators like GPUs, FPGAs, and programmable Network Interface Cards (SmartNICs) have become the bedrock of modern high-performance computing and machine learning. However, a glaring paradox exists in modern data centers: while these devices possess massive theoretical compute capacity, they frequently sit idle or underutilized.

A single application rarely saturates an accelerator's resources. Whether due to small batch sizes, memory bandwidth bottlenecks, or serial CPU overhead, expensive hardware cycles are routinely wasted. To bridge this efficiency gap, a prominent research direction focuses on workload co-location and hardware virtualization—the practice of running multiple independent models, workers, or data-processing handlers concurrently on a single physical device.


The Root of Underutilization

To understand why co-location is necessary, we must first understand why accelerators run out of work. It is intuitive to assume that feeding a massive model or a high-throughput data stream to a GPU or SmartNIC would keep it fully occupied. In practice, architectural bottlenecks frequently starve the compute engines.

The Memory and Communication Gap

In deep learning, training or serving models with small batch sizes is often desirable to maintain learning quality or minimize user-facing latency. However, Koliousis et al. observe that small-batch training introduces a severe communication-to-computation bottleneck, leaving the underlying processor heavily underutilized.

Even when batch sizes are scaled up, systems hit a "knee-point" of diminishing returns. As Wang et al. demonstrate, large-batch Large Language Model (LLM) inference does not automatically become compute-bound; instead, it remains fundamentally bottlenecked by DRAM bandwidth saturation. This leaves significant portions of the GPU's compute cores and memory capacity completely untouched, a phenomenon also documented by Samsi et al., who note widespread under-utilization of GPU memory across distributed LLM workloads.

The Preprocessing and Pipeline Gap

Accelerators do not operate in a vacuum; they rely on host CPUs to feed them data. When a single worker process handles both data preprocessing (which is CPU-bound) and model execution (which is GPU-bound), the accelerator sits idle during the preprocessing phase. Granqvist et al. highlight this friction in federated learning simulations. Similarly, in sequential decoding pipelines for language models, hardware components experience low duty cycles because resources remain idle during the sequential transitions typical of autoregressive execution, as noted by McDanel et al..


The Mechanics of Co-location and Replication

The core insight of this research space is that if a single workload cannot saturate an accelerator, the system should dynamically pack multiple workloads onto the same device to fill the execution gaps.

Uniprocess Execution (Wasted Cycles):
[ Compute ] --(Idle/CPU Bound)-- [ Compute ] --(Idle/CPU Bound)--

Co-located Execution (Saturated Hardware):
Worker 1: [ Compute ] ----------------- [ Compute ] -----------------
Worker 2: ----------- [ Compute ] ----------------- [ Compute ] -----
Combined: [ Compute ] [ Compute ] ----- [ Compute ] [ Compute ] -----

Model Replication and Worker Stacking

For workloads with a small memory footprint, such as Small Language Models (SLMs), Recasens et al. show that partitioning a single accelerator to run multiple independent model replicas yields superior performance compared to simply increasing the batch size of a single instance. This strategy fills idle compute cycles and avoids the latency penalties of massive batches. To combat CPU-bound delays, Granqvist et al. demonstrate that stacking multiple worker processes on a single GPU ensures continuous hardware utilization during non-GPU overhead periods.

Statistical Multiplexing and Hybrid Serving

When managing diverse applications, workloads can be co-located to balance each other out. Li et al. introduce the concept of statistical multiplexing for deep learning serving. By distributing multiple models across a shared pool of accelerators, idle resources allocated to a quiet model can instantly absorb sudden traffic bursts hitting another model, drastically reducing queuing delays.

Alternatively, systems can pair entirely different classes of tasks. Qiao et al. design a system that co-serves latency-sensitive online requests alongside high-throughput offline batch tasks on the same model architecture. This allows the offline tasks to "harvest" idle GPU cycles at a fine-grained token or layer level without incurring the overhead of process switching or reloading weights into memory.


Virtualization and Hardware Partitioning

To make co-location practical in multi-tenant environments like the cloud, software frameworks require underlying virtualization mechanisms to slice physical hardware into isolated, manageable units.

As outlined by Varghese et al., early generations of accelerators lacked mature, production-level virtualization, preventing efficient multi-tenant sharing. Lightweight accelerator virtualization has emerged to allow multiple applications to share specialized hardware with minimal overhead. These virtualization layers can even bridge architectural divides, exposing physical GPU resources to virtual machines running across completely different CPU architectures, such as x86 and ARM, as highlighted by Laccetti et al..

Modern hardware features have increasingly codified these capabilities. For example, in data-intensive tasks like genomic alignment, Gulhan et al. leverage hardware-level multi-instance GPU (MIG) features and multi-process services (MPS) to run independent alignment tasks concurrently. If one task encounters an I/O bottleneck, the other parallel tasks keep the computational cores spinning, preventing hardware starvation.

To shield developers from the complexity of managing these hardware slices, automated orchestration toolkits have emerged:

  • INFaaS: Developed by Romero et al., this framework automatically manages the co-location of diverse model variants and applications on shared accelerators to optimize throughput and minimize operational costs.
  • FIRST: As detailed by Tanikanti et al., this toolkit brings cloud-like multi-tenant scheduling to high-performance computing (HPC) clusters, dynamically packing multiple distinct, smaller AI models onto shared GPU nodes.

The Frontier: Enforcing Isolation and Security

While co-location solves the utilization problem, it introduces a critical challenge: isolation. When multiple independent tenants or user-defined handlers execute custom logic on shared hardware, they inevitably compete for shared microarchitectural resources like caches, interconnects, and memory buses.

This problem is particularly acute in network-accelerated environments. Frameworks like sPIN, introduced by Hoefler et al., allow users to offload custom packet-processing handlers directly to the Network Interface Card (NIC) to bypass CPU bottlenecks. However, as Besta points out, managing resource contention and enforcing Quality of Service (QoS) between independent tenants on shared programmable network hardware remains an open challenge.

Without strict hardware-level isolation, one tenant's poorly written or malicious handler can degrade the performance of another tenant's workload. Schneider et al. and Jiang et al. emphasize that future research in programmable network interfaces and FPGAs must develop robust multi-tenancy mechanisms. The goal is to guarantee that executing concurrent, user-defined logic never compromises data privacy or performance predictability.

The evolution of this field demonstrates a clear trajectory: hardware accelerators are transitioning from isolated, single-purpose coprocessors into fully virtualized, multi-tenant utility resources capable of safely and efficiently maximizing every available clock cycle.

Key Foundational Work

Cloud Futurology (2019) — contributed 3 concept elements to this direction.
Guest Editorial for Hybrid Parallelism in New HPC Systems (2017) — contributed 2 concept elements to this direction.
CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hardware_level_handler_isolation_800638e4,
  title = {hardware-level handler isolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-level-handler-isolation/rdr_800638e4e93a400fb95434ccce2d2cfe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hardware_level_handler_isolation_800638e4_v1,
  title = {hardware-level handler isolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-level-handler-isolation/rdr_800638e4e93a400fb95434ccce2d2cfe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake