In the early days of cloud computing, virtualization revolutionized hardware utilization by decoupling software from physical machines. As outlined by Hai Jin et al. in From Grid Computing to Cloud Computing: Experiences on Virtualization Technology, early virtualization relied on partitioning memory and I/O channels to allow multiple guest operating systems to share a single physical machine. For standard CPUs, this sharing is often managed temporally via time-slicing: the operating system rapidly switches between tasks, giving each process full access to the processor for a brief window.
However, as modern workloads have shifted toward massive parallel accelerators—such as Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Field Programmable Gate Arrays (FPGAs)—traditional temporal multiplexing has run into a wall.
When high-performance accelerators are time-sliced, they suffer from severe context-switching overheads. For instance, in architectures with large local scratchpad memories, saving and restoring execution states introduces massive latency penalties. Furthermore, as noted by Adrien Gegout et al. in Towards an Efficient Containerized Cloud Gaming Platform, standard virtualization frameworks lack the flexibility to slice and schedule GPU resources with the same fine-grained control available to CPUs. This often leaves expensive hardware severely underutilized when a single task cannot saturate the device's entire compute or memory capacity.
To solve this, researchers have turned to spatial resource partitioning (also known as space-division multiplexing). Instead of dividing hardware across time, spatial partitioning divides the physical fabric of an accelerator—its streaming multiprocessors, processing cores, or memory channels—into distinct, isolated subsets that execute heterogeneous workloads simultaneously.
Temporal Multiplexing (Time-Slicing)
Time 1: [======= Task A (100% Hardware) =======]
Time 2: [======= Task B (100% Hardware) =======]
Spatial Multiplexing (Space-Division)
Time 1: [--- Task A (30%) ---][----- Task B (70%) -----]
Time 2: [--- Task A (30%) ---][----- Task B (70%) -----]
The Core Intuition: Compute-Memory Complementarity
The primary reason spatial partitioning yields massive performance gains is resource complementarity. Different phases of modern AI workloads bottleneck on entirely different hardware components.
Consider Large Language Model (LLM) serving, which consists of two distinct phases:
- The Prefill Phase: Processes the initial prompt. This phase is highly parallelizable and compute-bound, saturating the GPU's arithmetic logic units (ALUs) while leaving memory bandwidth underutilized.
- The Decode Phase: Generates text token-by-token. This phase is sequentially constrained and memory-bound, constantly fetching model weights from high-bandwidth memory (HBM) while leaving the GPU's compute units largely idle.
If these phases are executed sequentially on the same hardware, the GPU alternates between starving its compute engines and starving its memory bus.
In DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing, Lei Gao et al. demonstrate that by dynamically partitioning Streaming Multiprocessors (SMs) on a single GPU, prefill and decode tasks can run concurrently. The compute-heavy prefill task occupies one spatial partition, while the memory-heavy decode task occupies another. Because their bottlenecks are inverted, they fill each other's utilization gaps, maximizing overall throughput without requiring expensive inter-device data transfers.
A similar synergy exists in Multimodal Large Language Models (MLLMs) and agentic vision workflows. As shown by Zhicheng Li et al. in SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs, compute-heavy vision encoders and memory-hungry language decoders can be co-executed on partitioned GPU resources. This spatial isolation prevents kernels from competing for the same execution units, significantly reducing token generation latency.
Shengzhong Liu et al. apply this principle to real-time edge applications in Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization, slicing GPU SMs into disjoint subsets to eliminate kernel launch contention between vision and language stages.
Achieving Strict Performance Isolation
Beyond maximizing utilization, spatial partitioning is essential for multi-tenancy and Service Level Objective (SLO) guarantees. When multiple applications share an accelerator via software-level scheduling, they inevitably interfere with one another at the hardware level, causing unpredictable latency spikes (tail latency).
Spatial partitioning solves this by drawing hard physical boundaries between workloads:
- Neural Processing Units (NPUs): In Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units, Dahu Feng et al. introduce spatial multi-tenancy for data-flow accelerators. By allocating distinct physical cores to different users, they bypass the massive context-switching overhead of saving and restoring large local scratchpad memories, ensuring predictable performance for every tenant.
- FPGAs: Traditional cloud FPGAs are time-sliced or managed via coarse software layers. In Enabling Efficient and Flexible FPGA Virtualization for Deep Learning in the Cloud, Shulin Zeng et al. propose space-division multiplexing, physically carving a single large FPGA fabric into independent hardware cores. This physical separation keeps performance deviations under heavy resource competition below 1%. Similarly, the authors of Architecture Support for FPGA Multi-tenancy in the Cloud utilize a "soft" Network-on-Chip (NoC) to divide an FPGA into logically isolated slots, vastly increasing device utilization.
- Solid-State Drives (SSDs): Resource contention is not limited to compute. In FlashBlox: Achieving Both Performance Isolation and Uniform Lifetime for Virtualized SSDs, Jian Huang et al. demonstrate that dedicating physical SSD channels and dies to individual tenants eliminates I/O interference, reducing 99th-percentile tail latency by up to 3.1$\times$ compared to software-isolated approaches.
- Many-Core CPUs: Even within many-core architectures like the Intel Xeon Phi, Zhaokui Li et al. show in Evaluating the Performance Impact of Multiple Streams on the MIC-based Heterogeneous Platform that dividing processing cores into independent spatial groups minimizes cache interference and resource contention.
Implementation Across the Stack
The mechanisms used to enforce spatial partitioning vary depending on the hardware and the target application.
Fine-Grained GPU Sharing
For deep learning workloads, frameworks must allocate precise fractions of a GPU's capacity. In Graft: Efficient Inference Serving for Hybrid Deep Learning with SLO Guarantees via DNN Re-alignment, Fangming Liu et al. assign specific percentages of hardware capacity to concurrent processes to optimize split-inference between mobile devices and edge servers. In educational or multi-user lab environments, Chunlin Chen demonstrates in A Practical Study on Building a Distributed Artificial Intelligence Experimental Teaching Platform Based on Traditional Laboratories that slicing physical GPUs into multiple virtual instances can double the concurrent user capacity of a computing facility.
Co-Executing Inference and Training
Edge devices often need to run real-time inference while simultaneously updating their models via background training. Doing this sequentially causes dropped frames, while unmanaged parallel execution causes severe lag.
To address this, Anurag Andhare et al. introduce Corun in Corun: Concurrent Inference and Continuous Training at the Edge for Cost-Efficient AI-Based Mobile Image Sensing, leveraging multiple CUDA streams to spatially multiplex a single GPU between live inference requests and continuous retraining jobs.
At the hardware level, Yoonsung Kim et al. propose DaCapo in DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics. DaCapo physically divides an accelerator into independent sub-engines, ensuring that background training tasks never interfere with the strict, real-time frame rate required for autonomous drone or vehicle inference.
Unmanaged Sharing (Contention)
[ Inference Kernel ] ──┐
├──> [ Shared Compute Units ] ──> Performance Drops
[ Training Kernel ] ──┘
Spatial Partitioning (Isolation)
[ Inference Kernel ] ─────> [ Sub-Accelerator Partition A ] ──> Stable Frame Rate
[ Training Kernel ] ─────> [ Sub-Accelerator Partition B ] ──> Background Updates
High-Throughput Networking
In Network Function Virtualization (NFV) systems, packet processing kernels must execute with minimal latency. Kai Zhang et al. present G-NET in G-NET: Effective GPU Sharing in NFV Systems, proving that assigning different network function kernels to distinct processing units simultaneously yields significantly higher throughput and lower latency than temporal sharing, making it highly effective for complex network service chains.
Conclusion
Spatial resource partitioning marks a paradigm shift in how we manage high-performance accelerators. By moving away from temporal time-slicing and physically dividing hardware into isolated spatial domains, modern systems eliminate context-switching overhead, guarantee performance isolation for multi-tenancy, and exploit the natural complementarity of compute- and memory-bound workloads. Whether applied to LLM serving in hyperscale datacenters or continuous learning on edge devices, spatial multiplexing ensures that every corner of an accelerator's silicon is working efficiently.
Key Foundational Work
A complete and efficient CUDA-sharing solution for HPC clusters (2014) — contributed 3 concept elements to this direction.
From Grid Computing to Cloud Computing: Experiences on Virtualization Technology (2010) — contributed 1 concept elements to this direction.