Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models

Yonggan Fu, Xin Dong, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan Celine Lin, Pavlo Molchanov

TL;DR

NVIDIA's Nemotron-Flash shifts the paradigm of Small Language Model (SLM) design from minimizing parameter counts to directly optimizing for real-device latency and throughput. By combining an automated evolutionary search for hybrid operators (DeltaNet, Mamba2, and Attention) with an augmented scaling law for depth-width ratios, the authors deliver a family of SLMs that significantly outperform state-of-the-art models, achieving up to 1.9× lower latency and 45.6× higher throughput than comparable baselines.

What Problem Does This Paper Address?

When deploying language models on resource-constrained edge devices or latency-sensitive applications, smaller is generally assumed to be better. Consequently, existing SLM design heavily prioritizes parameter efficiency—reducing the total number of parameters to fit a specific budget.

However, the authors point out a critical flaw: parameter efficiency does not translate into proportional real-device speedups on modern hardware accelerators like GPUs and TPUs. Many popular SLMs default to "deep-thin" architectures (many layers, small hidden dimensions) because they look good on paper parameter-wise. In practice, these architectures often suffer from hardware underutilization and suboptimal latency-accuracy trade-offs. Furthermore, while efficient attention alternatives (like Mamba or linear attention) exist, combining them into hybrid architectures has historically relied on slow, manual trial-and-error.

Methods

The authors tackle latency-optimal SLM design through a three-pronged approach spanning architecture, search, and training:

  1. Augmented Scaling Laws for Depth-Width Ratios: Instead of treating model size as a single variable, the authors decouple it into depth ($D$) and width ($W$) within the language modeling scaling law: $$\(D, W, N) = \_0 + aD^{-\alpha} + bW^{-\beta} + cN^{-\gamma}$$ This allows practitioners to profile configurations and select the exact "sweet-spot" depth-width ratio that minimizes loss for a given hardware latency constraint.

  2. Evolutionary Search for Hybrid Operators: The paper introduces an aging evolutionary search framework to automatically discover the best interleaved combinations of operators. The search space includes standard Attention, Mamba2, and DeltaNet (a linear attention variant). To avoid the massive compute cost of training every candidate to convergence, they use short-training perplexity (PPL) as a proxy, demonstrating an 88.8% Spearman rank correlation with final performance.

  3. Weight Normalization (wnorm): To boost training efficiency, the authors introduce a technique that projects weight matrices onto a unit norm sphere after each iteration. This eliminates radial weight growth, forces purely angular updates, and maintains a higher effective learning rate during the later stages of training.

Datasets

  • Pre-training & Exploration: Initial architecture exploration and scaling law fitting were conducted on 100B tokens sampled from the Smollm-corpus.
  • Final Production Training: The Nemotron-Flash-1B and 3B models were trained on 4.5T tokens. The training pipeline started with the Zyda2 dataset, followed by a high-quality blend including Climb-Mix, Smollm-corpus, MegaMath, and a proprietary dataset rich in math and code.
  • Evaluation: Models were benchmarked across 16 downstream tasks using the lm-evaluation-harness, covering MMLU, commonsense reasoning (PIQA, Hellaswag, etc.), math (GSM8K), coding (HumanEval, MBPP), and long-context recall (Ruler benchmark).

Key Insights and Ideas

  • The "Deep-Thin" Trap: While deeper, thinner models achieve better accuracy per parameter, they are often highly inefficient on actual hardware for small batch sizes. Latency-optimal depth-width ratios actually scale up with the target latency budget.
  • Interleaved Synergy: The automated search converged on a highly specific macro-architecture: interleaving DeltaNet-FFN-Mamba2-FFN and Attention-FFN-Mamba2-FFN blocks. DeltaNet and Mamba2 possess complementary memory mechanisms that narrow the performance gap of individual operators when paired.
  • Sparse Attention is Enough for Recall: Nemotron-Flash-1B and 3B achieve top-tier recall accuracy despite containing only 2 and 3 full-attention layers, respectively. This confirms that maintaining a full KV cache across every single layer is unnecessary for strong context retrieval.

Key Results

Nemotron-Flash establishes a new state-of-the-art accuracy-efficiency frontier for small language models:

Visualizing the accuracy-latency and accuracy-throughput trade-offs of Nemotron-Flash against SOTA SLMs.
  • Nemotron-Flash-3B achieves +2.0% higher average accuracy, 1.7× lower latency, and 6.4× higher throughput compared to Qwen2.5-3B.
  • Nemotron-Flash-1B delivers +5.5% higher average accuracy, 1.9× lower latency, and 45.6× higher throughput than Qwen3-0.6B.
  • Weight Normalization Impact: Applying wnorm consistently improved downstream commonsense reasoning accuracy by an average of +1.20% and reduced perplexity across multiple model families without the heavy 20%+ compute overhead of alternative methods like nGPT.

Limitations and Future Work

The authors note that their evolutionary search framework primarily relies on standard language modeling perplexity as its short-training proxy. While highly effective for general capabilities, this proxy might not fully capture specialized long-context retrieval performance during the initial search phase. Additionally, the current search space focuses strictly on macro-architecture configurations; extending the framework to optimize fine-grained micro-architectural factors remains an open direction for future research.

Summary updated

Abstract

Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. Specifically, we identify two central architectural factors: depth-width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth-width ratios, with the key finding that although deep-thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy-latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy-latency frontier. In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5% average accuracy, 1.3x/1.9x lower latency, and 18.7x/45.6x higher throughput compared to Qwen3-1.7B/0.6B, respectively.
BibTeX
@inproceedings{fu2025nemotron,
  author = {Yonggan Fu and Xin Dong and Shizhe Diao and Matthijs Van keirsbilck and Hanrong Ye and Wonmin Byeon and Yashaswi Karnati and Lucas Liebenwein and Hannah Zhang and Nikolaus Binder and Maksim Khadkevich and Alexander Keller and Jan Kautz and Yingyan Celine Lin and Pavlo Molchanov},
  title = {{Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=KTDAbnFsQj},
}
Figure