Spatial Resource Partitioning for Concurrent Accelerator Execution

30 concepts21 papers1 proposal

In the early days of cloud computing, virtualization revolutionized hardware utilization by decoupling software from physical machines. As outlined by Hai Jin et al. in From Grid Computing to Cloud Computing: Experiences on Virtualization Technology, early virtualization relied on partitioning memory and I/O channels to allow multiple guest operating systems to share a single physical machine. For standard CPUs, this sharing is often managed temporally via time-slicing: the operating system rapidly switches between tasks, giving each process full access to the processor for a brief window.

However, as modern workloads have shifted toward massive parallel accelerators—such as Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Field Programmable Gate Arrays (FPGAs)—traditional temporal multiplexing has run into a wall.

When high-performance accelerators are time-sliced, they suffer from severe context-switching overheads. For instance, in architectures with large local scratchpad memories, saving and restoring execution states introduces massive latency penalties. Furthermore, as noted by Adrien Gegout et al. in Towards an Efficient Containerized Cloud Gaming Platform, standard virtualization frameworks lack the flexibility to slice and schedule GPU resources with the same fine-grained control available to CPUs. This often leaves expensive hardware severely underutilized when a single task cannot saturate the device's entire compute or memory capacity.

To solve this, researchers have turned to spatial resource partitioning (also known as space-division multiplexing). Instead of dividing hardware across time, spatial partitioning divides the physical fabric of an accelerator—its streaming multiprocessors, processing cores, or memory channels—into distinct, isolated subsets that execute heterogeneous workloads simultaneously.

Temporal Multiplexing (Time-Slicing)
Time 1: [======= Task A (100% Hardware) =======]
Time 2: [======= Task B (100% Hardware) =======]

Spatial Multiplexing (Space-Division)
Time 1: [--- Task A (30%) ---][----- Task B (70%) -----]
Time 2: [--- Task A (30%) ---][----- Task B (70%) -----]

The Core Intuition: Compute-Memory Complementarity

The primary reason spatial partitioning yields massive performance gains is resource complementarity. Different phases of modern AI workloads bottleneck on entirely different hardware components.

Consider Large Language Model (LLM) serving, which consists of two distinct phases:

  1. The Prefill Phase: Processes the initial prompt. This phase is highly parallelizable and compute-bound, saturating the GPU's arithmetic logic units (ALUs) while leaving memory bandwidth underutilized.
  2. The Decode Phase: Generates text token-by-token. This phase is sequentially constrained and memory-bound, constantly fetching model weights from high-bandwidth memory (HBM) while leaving the GPU's compute units largely idle.

If these phases are executed sequentially on the same hardware, the GPU alternates between starving its compute engines and starving its memory bus.

In DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing, Lei Gao et al. demonstrate that by dynamically partitioning Streaming Multiprocessors (SMs) on a single GPU, prefill and decode tasks can run concurrently. The compute-heavy prefill task occupies one spatial partition, while the memory-heavy decode task occupies another. Because their bottlenecks are inverted, they fill each other's utilization gaps, maximizing overall throughput without requiring expensive inter-device data transfers.

A similar synergy exists in Multimodal Large Language Models (MLLMs) and agentic vision workflows. As shown by Zhicheng Li et al. in SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs, compute-heavy vision encoders and memory-hungry language decoders can be co-executed on partitioned GPU resources. This spatial isolation prevents kernels from competing for the same execution units, significantly reducing token generation latency.

Shengzhong Liu et al. apply this principle to real-time edge applications in Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization, slicing GPU SMs into disjoint subsets to eliminate kernel launch contention between vision and language stages.


Achieving Strict Performance Isolation

Beyond maximizing utilization, spatial partitioning is essential for multi-tenancy and Service Level Objective (SLO) guarantees. When multiple applications share an accelerator via software-level scheduling, they inevitably interfere with one another at the hardware level, causing unpredictable latency spikes (tail latency).

Spatial partitioning solves this by drawing hard physical boundaries between workloads:


Implementation Across the Stack

The mechanisms used to enforce spatial partitioning vary depending on the hardware and the target application.

Fine-Grained GPU Sharing

For deep learning workloads, frameworks must allocate precise fractions of a GPU's capacity. In Graft: Efficient Inference Serving for Hybrid Deep Learning with SLO Guarantees via DNN Re-alignment, Fangming Liu et al. assign specific percentages of hardware capacity to concurrent processes to optimize split-inference between mobile devices and edge servers. In educational or multi-user lab environments, Chunlin Chen demonstrates in A Practical Study on Building a Distributed Artificial Intelligence Experimental Teaching Platform Based on Traditional Laboratories that slicing physical GPUs into multiple virtual instances can double the concurrent user capacity of a computing facility.

Co-Executing Inference and Training

Edge devices often need to run real-time inference while simultaneously updating their models via background training. Doing this sequentially causes dropped frames, while unmanaged parallel execution causes severe lag.

To address this, Anurag Andhare et al. introduce Corun in Corun: Concurrent Inference and Continuous Training at the Edge for Cost-Efficient AI-Based Mobile Image Sensing, leveraging multiple CUDA streams to spatially multiplex a single GPU between live inference requests and continuous retraining jobs.

At the hardware level, Yoonsung Kim et al. propose DaCapo in DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics. DaCapo physically divides an accelerator into independent sub-engines, ensuring that background training tasks never interfere with the strict, real-time frame rate required for autonomous drone or vehicle inference.

Unmanaged Sharing (Contention)
[ Inference Kernel ] ──┐
                       ├──> [ Shared Compute Units ] ──> Performance Drops
[ Training Kernel  ] ──┘

Spatial Partitioning (Isolation)
[ Inference Kernel ] ─────> [ Sub-Accelerator Partition A ] ──> Stable Frame Rate
[ Training Kernel  ] ─────> [ Sub-Accelerator Partition B ] ──> Background Updates

High-Throughput Networking

In Network Function Virtualization (NFV) systems, packet processing kernels must execute with minimal latency. Kai Zhang et al. present G-NET in G-NET: Effective GPU Sharing in NFV Systems, proving that assigning different network function kernels to distinct processing units simultaneously yields significantly higher throughput and lower latency than temporal sharing, making it highly effective for complex network service chains.

Conclusion

Spatial resource partitioning marks a paradigm shift in how we manage high-performance accelerators. By moving away from temporal time-slicing and physically dividing hardware into isolated spatial domains, modern systems eliminate context-switching overhead, guarantee performance isolation for multi-tenancy, and exploit the natural complementarity of compute- and memory-bound workloads. Whether applied to LLM serving in hyperscale datacenters or continuous learning on edge devices, spatial multiplexing ensures that every corner of an accelerator's silicon is working efficiently.

Key Foundational Work

A complete and efficient CUDA-sharing solution for HPC clusters (2014) — contributed 3 concept elements to this direction.
From Grid Computing to Cloud Computing: Experiences on Virtualization Technology (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatial_hardware_partitioning_91d64f8c,
  title = {spatial hardware partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-hardware-partitioning/rdr_91d64f8c7ceb4fb2ab8a88bc0530b0d1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatial_hardware_partitioning_91d64f8c_v1,
  title = {spatial hardware partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-hardware-partitioning/rdr_91d64f8c7ceb4fb2ab8a88bc0530b0d1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake