High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion

Aditya Chatterjee, Sanjay Singh Chauhan, Mayank Jain, Arushi Dhingra, Sahin Chowdhury, Rahul MittalarXiv 2026 · arXiv

TL;DR

NVIDIA researchers have developed a "regime-aware" serving engine for Nemotron 3.5 Transcribe that dynamically switches between massive "megakernels" and chains of fused kernels based on real-time workload size. This approach achieves up to 4.4x higher throughput than vLLM on H100 GPUs and serves 8x more concurrent requests within a strict 400ms latency budget, all while maintaining state-of-the-art transcription accuracy.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) at scale is a balancing act between latency and throughput. Nemotron 3.5 Transcribe is a complex, hybrid model combining a Parallel Expert Encoder (ASR + diarization) with a Mamba-Transformer Mixture-of-Experts (MoE) backbone.

Standard inference frameworks like vLLM often struggle with such models because they execute operations as separate kernels, leading to significant overhead from intermediate memory traffic and synchronization. While "megakernels"—which fuse an entire region of a model into a single GPU launch—can eliminate this overhead, they aren't a silver bullet. A megakernel that is lightning-fast for a small batch might become a bottleneck for a large one because it can't saturate the GPU's compute units as effectively as a chain of specialized kernels. The authors tackle the challenge of choosing the optimal "fusion granularity" on the fly as batch sizes fluctuate during continuous serving.

Methods

The core innovation is the Regime-Aware Serving Engine. Instead of picking one fusion strategy, the engine uses two levels of implementation for key model regions:

  1. Persistent Megakernels: These run a whole model region (like the ASR encoder) in a single launch. They are "stable" (always used) for some components and "receding" for others, meaning they are only used when the workload is below a specific "crossover" threshold.
  2. Fused Kernel Chains: For larger workloads, the engine switches to a chain of custom kernels (using CUTLASS and Programmatic Dependent Launch) that retain independent launch configurations to better utilize the GPU's parallel resources.

The engine measures these crossover points offline. For example, the ASR encoder uses a megakernel for passes under 384 tokens but switches to fused kernels for larger batches. Additionally, the engine optimizes the Mamba-Transformer-MoE backbone by mixing "prefill" (processing new audio) and "decode" (generating text) rows in the same step, ensuring the GPU's weights are read only once per iteration.

Datasets

The authors evaluated their engine using:

  • NVIDIA English Bias corpus: A primary benchmark of 4,820 recordings (15.36 hours) covering various domains (telecom, technical, etc.) and accents.
  • LibriSpeech: Specifically the test-clean and test-other splits (5,559 recordings) to validate the engine on a standard public corpus.
  • Calibration Set: 512 LibriSpeech development utterances used to tune "opt-in" optimizations like MoE layer skipping.

Key Insights and Ideas

  • The "Receding" Megakernel: The most important "aha" moment is that megakernels lose their performance edge as batch sizes grow. By identifying "megakernel-receding" architectures, the authors proved that a hybrid approach—switching regimes at measured crossovers—beats using either megakernels or fused kernels alone by up to 23%.
  • Prefill-Dominated Serving: Unlike standard LLM chat, which is often bottlenecked by token generation (decoding), ASR is dominated by the initial audio processing (prefill). The engine is specifically tuned to handle this prefill-heavy mix efficiently.
  • The Energy-Roofline Model: On power-capped GPUs (like the 400W H100 NVL), the authors found that traditional "roofline" performance models (where memory access hides under compute) break down. Instead, memory and compute compete for the same "watts," meaning their execution times add together rather than overlapping. This insight drove them to focus on reducing total bytes moved and operations performed rather than just improving overlap.

Key Results

  • Throughput: On an H100 SXM GPU, the engine reached 4,345x real-time throughput, a 1.39x improvement over a tuned vLLM baseline. On the newer B200 GPU, it hit 6,367x real-time.
  • Latency: In a 400ms p95 end-to-end completion time budget, the engine served 8x as many concurrent requests as vLLM.
  • Accuracy: The optimizations are "accuracy-preserving," maintaining a Word Error Rate (WER) of 1.91% to 1.93%, virtually identical to the vLLM reference.
  • Extreme Throughput Mode: By enabling "opt-in" features like greedy MoE layer skipping, they could boost throughput by another 13-16% for offline jobs that can tolerate a slightly higher WER (~3.4%).

Limitations and Future Work

  • Offline Crossover Determination: Currently, the thresholds for switching between megakernels and fused kernels are determined offline. The authors suggest that an online approach to determine these crossovers dynamically would be a valuable next step.
  • Generalization: While the "regime-aware" lessons are applicable to other hybrid models, the specific kernels are highly optimized for the Nemotron 3.5 architecture.
  • Dynamic Dominance: The paper notes that as different model components (like MoE routing) become more dominant in future architectures, the optimal granularity of fusion will likely change, requiring new performance models.

Summary updated

Abstract

Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR leaderboards. Serving it efficiently takes more than fast operators: continuous batching changes the amount and mix of work at every iteration, and the best fusion granularity changes with it. We present a serving engine that, for every pass, chooses between persistent megakernels and chains of fused kernels at crossovers measured offline. On an H100 SXM GPU, throughput of our accuracy-preserving serving peaks at 4,345x real time, 1.39x the peak of tuned vLLM, and is up to 4.39x faster at low concurrency; on B200 it peaks at 6,367x, 1.64x vLLM at the highest concurrency we measured. Word error rate (WER) is 1.91 to 1.93% against 1.88 to 1.93% for vLLM. Within a 400 ms budget for p95 end-to-end completion time, it serves 8x as many concurrent requests. Choosing per pass beats either level alone: 23% higher throughput than always using megakernels at 128 concurrent requests, and 9% higher than always using fused kernels at 8. Opt-in switches, including greedy MoE layer skipping, add 13 to 16% throughput on B200 for offline jobs that can accept higher WER.
BibTeX
@misc{chatterjee2026high,
  author = {Aditya Chatterjee and Sanjay Singh Chauhan and Mayank Jain and Arushi Dhingra and Sahin Chowdhury and Rahul Mittal},
  title = {{High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion}},
  year = {2026},
  eprint = {2611.20095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20095},
}
Figure