Traditional GPU programming models treat kernels as monolithic grids of threads that occupy the hardware until completion. While this maximizes throughput for massive, uniform workloads, it creates a "granularity gap" when dealing with irregular data, real-time constraints, or shared environments. Modern research has shifted toward decomposing these monolithic kernels into finer units—ranging from thread blocks to sub-warp groups—to improve hardware utilization and enable precise resource control.
Breaking the Monolith: Thread-Block Level Slicing
The primary challenge in shared GPU environments is "head-of-line blocking," where a long-running, low-priority kernel prevents a high-priority task from starting. To solve this, researchers have proposed partitioning large compute grids into smaller, discrete sub-kernels or "slices."
Wei Zhao et al. demonstrate that scheduling at the thread-block level provides an optimal balance between control and simplicity. By executing these slices sequentially, their system, Tally, creates frequent scheduling checkpoints where high-priority tasks can be interleaved with minimal latency. Similarly, Jianlong Zhong shows that grouping thread blocks at the assembly level allows for dynamic "slicing," enabling the co-scheduling of complementary tasks—such as pairing a compute-heavy slice with a memory-heavy one—to boost overall throughput.
This concept is taken further by Coppock et al., who decompose kernels into "atoms" scheduled at the level of individual hardware clusters. This fine-grained preemption allows the GPU to pause execution between discrete units, slashing tail latencies without requiring code changes from the user.
Sub-Warp Partitioning for Irregular Workloads
While thread-block slicing addresses inter-kernel scheduling, intra-kernel efficiency often suffers when the workload is irregular. In the standard SIMT (Single Instruction, Multiple Threads) model, a "warp" of 32 threads executes in lockstep. If only a few threads have work to do, the rest of the warp sits idle.
To mitigate this, Park et al. propose splitting CUDA warps into smaller subwarps. In applications like DNA sequence alignment, this allows for more granular processing of dynamic programming tables, particularly during phases where thread utilization is naturally low. By balancing parallelism at the sub-warp level, they effectively improve workload distribution across the hardware.
This strategy is also vital for graph theory applications. Almasri et al. utilize sub-warp partitioning for K-clique counting, arguing that partitioning work into groups smaller than a warp better matches computational needs to hardware capabilities. This finer granularity ensures that the GPU's massive parallel resources are not wasted on divergent or sparse data structures.
Exploiting Micro-Bubbles and Elasticity
Fine-grained scheduling also allows systems to "fill the gaps" in GPU execution timelines, often referred to as bubbles. In large-scale model training, communication between GPUs often leaves the computation units idle for milliseconds. Lin et al. show that by decomposing model layers into fine-grained kernels, they can exploit these "micro-bubbles" to perform encoder computations that would otherwise be stalled.
For real-time systems, Zhihe Zhao et al. introduce the concept of "elastic kernels." By transforming standard kernels into resizable shards, their framework, Miriam, can fill idle execution cycles with low-priority tasks that do not interfere with the timing of critical workloads. This intra-kernel specialization is echoed by Shulai Zhang et al., who move task scheduling directly into GPU thread blocks to eliminate the execution gaps typically seen in multi-stream models.
Mapping to the Finest Granularity
The success of these scheduling techniques relies on identifying the smallest possible unit of work within a specific domain. Across various fields, the trend is to move away from coarse-grained parallelization:
- Logic Simulation: Bo D. Wang et al. achieve significant speedups by mapping simulation to individual gates or pins rather than larger circuit blocks.
- Structural Biology: Kimanius et al. accelerate cryo-EM processing by treating individual pixels and orientations as independent tasks.
- Physics Simulation: Loeb et al. demonstrate that parallelizing finite element calculations at the level of individual degrees of freedom, rather than the element level, maximizes the utilization of GPU cores.
By shifting the focus from "how many threads can we run" to "how small can we make the schedulable unit," these researchers have unlocked new levels of efficiency and responsiveness in GPU computing.
Key Foundational Work
Distributed time, conservative parallel logic simulation on GPUs (2010) — contributed 1 concept elements to this direction.
Data-parallel rasterization of micropolygons with defocus and motion blur (2009) — contributed 1 concept elements to this direction.