Fine-Grained GPU Scheduling and Subwarp Partitioning

34 concepts26 papers1 proposal

Traditional GPU programming models treat kernels as monolithic grids of threads that occupy the hardware until completion. While this maximizes throughput for massive, uniform workloads, it creates a "granularity gap" when dealing with irregular data, real-time constraints, or shared environments. Modern research has shifted toward decomposing these monolithic kernels into finer units—ranging from thread blocks to sub-warp groups—to improve hardware utilization and enable precise resource control.

Breaking the Monolith: Thread-Block Level Slicing

The primary challenge in shared GPU environments is "head-of-line blocking," where a long-running, low-priority kernel prevents a high-priority task from starting. To solve this, researchers have proposed partitioning large compute grids into smaller, discrete sub-kernels or "slices."

Wei Zhao et al. demonstrate that scheduling at the thread-block level provides an optimal balance between control and simplicity. By executing these slices sequentially, their system, Tally, creates frequent scheduling checkpoints where high-priority tasks can be interleaved with minimal latency. Similarly, Jianlong Zhong shows that grouping thread blocks at the assembly level allows for dynamic "slicing," enabling the co-scheduling of complementary tasks—such as pairing a compute-heavy slice with a memory-heavy one—to boost overall throughput.

This concept is taken further by Coppock et al., who decompose kernels into "atoms" scheduled at the level of individual hardware clusters. This fine-grained preemption allows the GPU to pause execution between discrete units, slashing tail latencies without requiring code changes from the user.

Sub-Warp Partitioning for Irregular Workloads

While thread-block slicing addresses inter-kernel scheduling, intra-kernel efficiency often suffers when the workload is irregular. In the standard SIMT (Single Instruction, Multiple Threads) model, a "warp" of 32 threads executes in lockstep. If only a few threads have work to do, the rest of the warp sits idle.

To mitigate this, Park et al. propose splitting CUDA warps into smaller subwarps. In applications like DNA sequence alignment, this allows for more granular processing of dynamic programming tables, particularly during phases where thread utilization is naturally low. By balancing parallelism at the sub-warp level, they effectively improve workload distribution across the hardware.

This strategy is also vital for graph theory applications. Almasri et al. utilize sub-warp partitioning for K-clique counting, arguing that partitioning work into groups smaller than a warp better matches computational needs to hardware capabilities. This finer granularity ensures that the GPU's massive parallel resources are not wasted on divergent or sparse data structures.

Exploiting Micro-Bubbles and Elasticity

Fine-grained scheduling also allows systems to "fill the gaps" in GPU execution timelines, often referred to as bubbles. In large-scale model training, communication between GPUs often leaves the computation units idle for milliseconds. Lin et al. show that by decomposing model layers into fine-grained kernels, they can exploit these "micro-bubbles" to perform encoder computations that would otherwise be stalled.

For real-time systems, Zhihe Zhao et al. introduce the concept of "elastic kernels." By transforming standard kernels into resizable shards, their framework, Miriam, can fill idle execution cycles with low-priority tasks that do not interfere with the timing of critical workloads. This intra-kernel specialization is echoed by Shulai Zhang et al., who move task scheduling directly into GPU thread blocks to eliminate the execution gaps typically seen in multi-stream models.

Mapping to the Finest Granularity

The success of these scheduling techniques relies on identifying the smallest possible unit of work within a specific domain. Across various fields, the trend is to move away from coarse-grained parallelization:

  • Logic Simulation: Bo D. Wang et al. achieve significant speedups by mapping simulation to individual gates or pins rather than larger circuit blocks.
  • Structural Biology: Kimanius et al. accelerate cryo-EM processing by treating individual pixels and orientations as independent tasks.
  • Physics Simulation: Loeb et al. demonstrate that parallelizing finite element calculations at the level of individual degrees of freedom, rather than the element level, maximizes the utilization of GPU cores.

By shifting the focus from "how many threads can we run" to "how small can we make the schedulable unit," these researchers have unlocked new levels of efficiency and responsiveness in GPU computing.

Key Foundational Work

Distributed time, conservative parallel logic simulation on GPUs (2010) — contributed 1 concept elements to this direction.
Data-parallel rasterization of micropolygons with defocus and motion blur (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subwarp_scheduling_91e8910b,
  title = {subwarp scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subwarp-scheduling/rdr_91e8910b4541491bbfa205ad70cd9f29},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subwarp_scheduling_91e8910b_v1,
  title = {subwarp scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subwarp-scheduling/rdr_91e8910b4541491bbfa205ad70cd9f29/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake