TL;DR
This paper challenges the "bigger is better" approach to kernel fusion in GPU inference. The authors demonstrate that a single monolithic "megakernel" is often suboptimal; instead, the best performance comes from strategically "cutting" a model into multiple specialized megakernels. They introduce an automated planner that predicts these optimal boundaries using first-principles simulation, achieving up to a 4.05x speedup in time-to-first-audio for streaming models.
What Problem Does This Paper Address?
In the world of real-time streaming inference (like Text-to-Speech or LLMs), two metrics reign supreme: Time to First Token/Audio (TTFT/TTFA) and Throughput. To minimize latency, developers often use "megakernels"—large, persistent GPU kernels that fuse multiple operations to avoid the overhead of launching thousands of tiny individual kernels.
However, there is a catch: a monolithic megakernel forces every operation inside it to share the same hardware configuration (like thread-block shapes and register limits). This "one-size-fits-all" approach often leaves performance on the table because different phases of a model (e.g., the prefill vs. the autoregressive decode) have vastly different resource needs. The challenge is deciding exactly where to "cut" the model—which parts should be fused together and which should remain separate—without the prohibitively expensive process of manually implementing and testing every possible combination.
Methods
The authors developed an automated Boundary Planner that identifies the best megakernel composition without running the full model end-to-end. The process involves three layers of modeling:
- Region Graph (rGraph): Identifies logical groups of operations (Prefill, Producer, Consumer) and their dependencies.
- Tile Graph (tGraph): A fine-grained model of how data moves between GPU "tiles." It accounts for hardware-level details like weight prefetching and on-chip data forwarding.
- Event Graph (evGraph): Simulates the entire execution timeline, accounting for how kernels compete for GPU resources (CTAs) and memory bandwidth.
The planner uses measurements from standard "conventional" execution and standalone benchmarks to score 81+ candidate designs. It models complex interactions, such as how "persistent" kernels hold onto GPU resources even while waiting for data, which can inadvertently block other kernels.
Datasets and Models
The framework was evaluated on an NVIDIA H100 NVL GPU across three state-of-the-art models:
- NH-TTS: An upcoming autoregressive Text-to-Speech model with a Nemotron-H backbone.
- Qwen3-TTS: A vLLM-based speech model.
- Llama 3.2 1B: A lightweight LLM.
The authors tested these models across various batch sizes (1 to 64) to see how the "optimal" cut changes as the workload scales.
Key Insights and Ideas
- The Specialization Gap: The "aha" moment is that the benefit of specialization often outweighs the cost of a kernel boundary. For NH-TTS, splitting the model into three megakernels allowed one stage to use 256-thread blocks while others used 512, leading to much better hardware utilization than a single monolith.
- The "Faster Kernel" Paradox: Counter-intuitively, making a single kernel faster can sometimes make the entire pipeline slower. If a faster kernel requires more thread blocks (CTAs), it might "starve" a co-running kernel of resources, creating a bottleneck elsewhere.
- Model Taxonomy: The authors categorize models based on how they respond to batch size:
- Megakernel-stable: Fusion always wins (e.g., NH-TTS).
- Megakernel-receding: Fusion helps at small batches, but conventional execution wins at large batches as overhead is amortized (e.g., Qwen3-TTS).
- Megakernel-advancing/averse: Predicted categories for models dominated by traffic or already compute-bound.
Key Results
The planner's predictions were remarkably accurate, selecting the measured fastest design in 13 out of 13 tested model-batch combinations.
- NH-TTS (Batch 32): The 3-megakernel design achieved a 4.05x speedup in TTFA compared to the conventional TensorRT baseline and was 21.4% faster than a single monolithic megakernel.
- Throughput: The same 3-megakernel design increased throughput (RTFx) by 28% over the monolith.
- Qwen3-TTS: The planner correctly identified that while 4 megakernels are best at batch 32, the model should revert to conventional execution at batch 48 and 64 to achieve the lowest latency.
Limitations and Future Work
While the results are impressive, the authors note a few limitations:
- Hardware Specificity: The current study focused on the NVIDIA H100 NVL; it's unclear how these "optimal cuts" translate to different GPU architectures (e.g., A100 or gaming GPUs).
- Implementation Contracts: The planner relies on "source-declared contracts"—developers must provide some metadata about how their kernels are implemented.
- Future Directions: The authors suggest exploring software-level scheduling to better manage interference between co-located kernels and investigating whether certain neural architectures are inherently more "megakernel-friendly" than others.
Summary updated