One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

Aditya Chatterjee, Anand Joseph X M, Sanjay Singh ChauhanarXiv 2026 · arXiv

TL;DR

This paper challenges the "bigger is better" approach to kernel fusion in GPU inference. The authors demonstrate that a single monolithic "megakernel" is often suboptimal; instead, the best performance comes from strategically "cutting" a model into multiple specialized megakernels. They introduce an automated planner that predicts these optimal boundaries using first-principles simulation, achieving up to a 4.05x speedup in time-to-first-audio for streaming models.

What Problem Does This Paper Address?

In the world of real-time streaming inference (like Text-to-Speech or LLMs), two metrics reign supreme: Time to First Token/Audio (TTFT/TTFA) and Throughput. To minimize latency, developers often use "megakernels"—large, persistent GPU kernels that fuse multiple operations to avoid the overhead of launching thousands of tiny individual kernels.

However, there is a catch: a monolithic megakernel forces every operation inside it to share the same hardware configuration (like thread-block shapes and register limits). This "one-size-fits-all" approach often leaves performance on the table because different phases of a model (e.g., the prefill vs. the autoregressive decode) have vastly different resource needs. The challenge is deciding exactly where to "cut" the model—which parts should be fused together and which should remain separate—without the prohibitively expensive process of manually implementing and testing every possible combination.

Methods

The authors developed an automated Boundary Planner that identifies the best megakernel composition without running the full model end-to-end. The process involves three layers of modeling:

  1. Region Graph (rGraph): Identifies logical groups of operations (Prefill, Producer, Consumer) and their dependencies.
  2. Tile Graph (tGraph): A fine-grained model of how data moves between GPU "tiles." It accounts for hardware-level details like weight prefetching and on-chip data forwarding.
  3. Event Graph (evGraph): Simulates the entire execution timeline, accounting for how kernels compete for GPU resources (CTAs) and memory bandwidth.

The planner uses measurements from standard "conventional" execution and standalone benchmarks to score 81+ candidate designs. It models complex interactions, such as how "persistent" kernels hold onto GPU resources even while waiting for data, which can inadvertently block other kernels.

The planner models various hardware mechanisms to predict performance accurately.

Datasets and Models

The framework was evaluated on an NVIDIA H100 NVL GPU across three state-of-the-art models:

  • NH-TTS: An upcoming autoregressive Text-to-Speech model with a Nemotron-H backbone.
  • Qwen3-TTS: A vLLM-based speech model.
  • Llama 3.2 1B: A lightweight LLM.

The authors tested these models across various batch sizes (1 to 64) to see how the "optimal" cut changes as the workload scales.

Key Insights and Ideas

  • The Specialization Gap: The "aha" moment is that the benefit of specialization often outweighs the cost of a kernel boundary. For NH-TTS, splitting the model into three megakernels allowed one stage to use 256-thread blocks while others used 512, leading to much better hardware utilization than a single monolith.
  • The "Faster Kernel" Paradox: Counter-intuitively, making a single kernel faster can sometimes make the entire pipeline slower. If a faster kernel requires more thread blocks (CTAs), it might "starve" a co-running kernel of resources, creating a bottleneck elsewhere.
  • Model Taxonomy: The authors categorize models based on how they respond to batch size:
  • Megakernel-stable: Fusion always wins (e.g., NH-TTS).
  • Megakernel-receding: Fusion helps at small batches, but conventional execution wins at large batches as overhead is amortized (e.g., Qwen3-TTS).
  • Megakernel-advancing/averse: Predicted categories for models dominated by traffic or already compute-bound.
Fig. 3. Best megakernel composition for NH-TTS at batch size 32

Key Results

The planner's predictions were remarkably accurate, selecting the measured fastest design in 13 out of 13 tested model-batch combinations.

  • NH-TTS (Batch 32): The 3-megakernel design achieved a 4.05x speedup in TTFA compared to the conventional TensorRT baseline and was 21.4% faster than a single monolithic megakernel.
  • Throughput: The same 3-megakernel design increased throughput (RTFx) by 28% over the monolith.
  • Qwen3-TTS: The planner correctly identified that while 4 megakernels are best at batch 32, the model should revert to conventional execution at batch 48 and 64 to achieve the lowest latency.
Fig. 2. TTFA and RTFx for NH-TTS at batch size 32 with the selected three- megakernel design

Limitations and Future Work

While the results are impressive, the authors note a few limitations:

  • Hardware Specificity: The current study focused on the NVIDIA H100 NVL; it's unclear how these "optimal cuts" translate to different GPU architectures (e.g., A100 or gaming GPUs).
  • Implementation Contracts: The planner relies on "source-declared contracts"—developers must provide some metadata about how their kernels are implemented.
  • Future Directions: The authors suggest exploring software-level scheduling to better manage interference between co-located kernels and investigating whether certain neural architectures are inherently more "megakernel-friendly" than others.

Summary updated

Abstract

We address a fundamental question: How far should kernel fusion extend and which boundaries are worth preserving? Unlike the general convention of using one monolith megakernel at lower batch sizes or supporting only a selective model component as megakernel and rest of the model as conventional graph, we show that the best configuration is a composition of multiple megakernels with no overlapping conventional graph component, or a fully conventional graph. We present a framework that selects megakernel regions in a streaming inference model from conventional-graph profiles and standalone kernel benchmarks, without running any candidate design end to end. It simulates each candidate's tile and event graphs to predict time to first output and selects the measured best design in all 13 combinations of model and batch size we tested. Across NH-TTS, Qwen3-TTS and Llama-1B, the best composition ranges from two to four megakernels to fully conventional execution and none is a single megakernel. We introduce four model categories (megakernel-stable/ receding/ advancing/ averse) that capture how a model's best megakernel composition changes with batch size. Using NH-TTS model on H100 NVL GPU, we demonstrate that preserving three specialized megakernels can improve both latency and throughput despite introducing additional kernel boundaries. At batch size 32, this design delivers 21.4% lower mean batch-p95 time to first audio (TTFA) and 28% higher throughput than the evaluated monolith megakernel. Against conventional TensorRT graph execution, it reduces engine-level mean batch TTFA p95 by 4.05x and increases throughput by 19.7%.
BibTeX
@misc{chatterjee2026one,
  author = {Aditya Chatterjee and Anand Joseph X M and Sanjay Singh Chauhan},
  title = {{One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference}},
  year = {2026},
  eprint = {2610.35336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35336},
}
Figure