Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational Agents

Nvidia, Adi Margolin, Aditya Malte, Aishwarya Chhabra, Alex Hodson, Ameya Sunil Mahabaleshwarkar, Ambrish Dantrey, Amy Shen, Andrii Skliar, Annamalai Chockalingam, Anusha Pant, Ashwath Aithal, Ashwin Kumar, Besmira Nushi, Bhavesh Pawar, Bryan Catanzaro, Bryan Yu, Chantal Hwang, Chenhan Yu, Chris Alexiuk, Conner Currie, Cynthia Gao, David Mosallanezhad, Dina Yared, Ehsan Hosseini Asl, Eileen Long, Erick Galinkin, Felix Friedmann, Giedrius Burachas, Gorkem Batmaz, Grzegorz Chlebus, Guillermo Siman, Haifeng Qian, Hao Guo, Hayley Ross, Hrishith Thadicherla, Hui Wang, Huiying Li, Ike Nnoli, Irina Filatova, Iris Liu, Ji-Eun Lee, Jian Leong, Jian Zhang, Jiaqi Zeng, Joey Conway, Jonathan Cohen, Jose Vergara, Julien Veron Vialard, Kari Briski, Katherine Luna, Kedar Pandurang Potdar, Kevin Ning, Khalil Ben Khaled, Krzysztof Pawelec, Laura Dykes, Leili Tavabi, Lily Lee, Linda Liu, Liwei Ding, Maka Dong, Manoj Kilaru, Maria Bagdasarova, Matthew Lee, Meredith Price, Micah Schaffer, Michael Boone, Michael Fukuyama, Michael Hasper, Nancy Agarwal, Negar Habibi, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pablo Ribalta, Payal Motwani, Pinky Xu, Prasoon Varshney, Pritam Biswas, Puhui Meng, Rameshwar Shivbhakta, Ran Zilberstein, Reka Kovacs, Richard Mazzarese, Rishabh Garg, Rita Fernandes Neves, Rochelle Woods, Ruby Ye, Ruvaid Virk, Ryan Hill, Saeid Motiian, Seph Mard, Shai Ben Nun, Shaona Ghosh, Sharad Yadav, Sudeep Sabnis, Suhas Hariharapura Sheshadri, Terry Kong, Thanh Do, Thinh Le, Udi Karpas, Viet Anh Trinh, Vinay Raman, Vinita Sharma, Warren Drones, Wenwen Gao, Whei Hsueh, Will Efstratis, Yoshi Suhara, Yunsheng Liu, Zhenming Wang, Zijia Chen, Zuhair AhmedarXiv 2026 · arXiv

TL;DR

NVIDIA’s Nemotron 3.5 Bolt E4B 4GB is a highly efficient small language model (SLM) optimized for fast, local conversational agents on Edge devices. By leveraging pure reinforcement learning (RL) post-training and speculative decoding, it achieves state-of-the-art accuracy in tool use and instruction following while maintaining a tiny 4GB memory footprint and delivering up to a 2.74× speedup in inference.

What Problem Does This Paper Address?

Building conversational agents for local deployment (e.g., voice assistants, robots, or IoT devices) presents a "trilemma" of constraints: intelligence, memory, and latency.

  1. Intelligence: The model must reliably follow instructions and use tools across multiple turns.
  2. Memory: It must fit within the limited VRAM of consumer GPUs or Edge modules like NVIDIA Jetson.
  3. Latency: For natural interaction, the "Time to First Token" (TTFT) and generation speed must be near-instant.

Existing models often struggle to balance these, either being too large for local VRAM or too verbose and slow for real-time voice interfaces. Nemotron 3.5 Bolt aims to maximize "intelligence per GB" specifically for these fast-response scenarios.

Methods

The researchers started with Gemma-4-E4B-it as a base and applied several novel post-training and deployment strategies:

  • Pure RL Post-Training: Unlike the standard pipeline that uses Supervised Fine-Tuning (SFT) followed by RL, the authors found that SFT actually regressed conversational quality and increased verbosity. They opted for pure RL post-training using RLVR (Reinforcement Learning from Verifiable Rewards) and RLHF via the NeMo-RL framework.
  • Dual-Mode Reasoning: The model is trained to operate in both reasoning (enable_thinking=true) and non-reasoning (enable_thinking=false) modes. Every training sample was used in both modes to ensure high accuracy even when the "chain-of-thought" is disabled for speed.
  • Verbosity Control: To prevent the "chatty" nature of many LLMs, they implemented group-relative length penalties. This forces the model to be concise—a critical feature for voice-based agents.
  • DFlash Speculative Decoding: The model is released with a DFlash drafter (a 360M parameter Transformer). This drafter predicts multiple future tokens that the main model then verifies in parallel, significantly boosting throughput.
  • Quantization: The model is optimized for local deployment using Q4_K_M quantization with importance-matrix (imatrix) calibration, ensuring minimal accuracy loss at 4-bit precision.

Datasets

The model was post-trained on a diverse blend of environments (see Table 1 below) focusing on multi-turn tool use, instruction hierarchy, and structured outputs.

Environments and source datasets used in post-training

Evaluation spanned 21 benchmarks, including:

  • Conversational Proficiency: EQ-Bench v3, MultiChallenge, and RoleMRC.
  • Tool Use: BFCL-v3, ToolSandbox, and a new multi-turn extension called When2Call-MT.
  • Internal Brevity Benchmark: Used to measure the model's ability to provide high-quality, concise answers suitable for spoken conversation.

Key Insights and Ideas

  • Conciseness is a Feature, Not a Bug: In the "Smart and Fast" quadrant, Bolt E4B stands out by being significantly less verbose than competitors like Qwen3.5-4B while maintaining higher accuracy. For example, on reasoning tasks, it generates roughly 1/5th the tokens of Qwen while achieving better scores.
  • The SFT Regression: The discovery that SFT on diverse "teacher" data can hurt conversational quality for small models is a significant finding for researchers looking to optimize SLMs.
  • Multimodal Resilience: Despite being post-trained only on text, the model retains the vision and audio capabilities of its base model. This suggests that high-quality text RL doesn't necessarily "break" the underlying multimodal alignments.
Accuracy vs. Verbosity trade-off across benchmarks

Key Results

  • Performance: Bolt E4B achieved the highest average accuracy across 21 benchmarks compared to Gemma-4-E4B-it and Qwen3.5-4B.
  • Speed: With the DFlash drafter, the model achieves 2.74× the per-user decode speed at concurrency 1 on an RTX PRO 6000.
  • Efficiency: As shown in Figure 1, Bolt E4B occupies the "Smart and Fast" top-left corner, offering a superior balance of benchmark scores and TTFT compared to much larger models.
Benchmark score vs. Time to First Token (TTFT)
  • Live Testing: The model was successfully tested in real-time scenarios, including controlling a robot (Reachy) and acting as an NPC in Minecraft, demonstrating its ability to handle grounding and tool calls autonomously.

Reachy robot live testing Minecraft NPC agent testing

Limitations and Future Work

  • Vision Degradation: The authors noted a slight but consistent drop in vision performance (e.g., on MMMU-Pro Vision) compared to the base model, likely due to the text-only RL phase.
  • Audio Constraints: Evaluation on audio benchmarks was limited by a 30-second input window; many real-world conversations exceed this.
  • Language Support: While the model retains some multilingual capabilities, the post-training was English-only. Future work aims to restore and improve vision capabilities and expand multilingual support.

Summary updated

Abstract

We introduce Nemotron 3.5 Bolt E4B 4GB, an open and efficient small language model further post-trained from Gemma-4-E4B-it and optimized for speed and intelligence per GB of memory used for fast conversational agents. The model carries forward Gemma's strong conversational capabilities and further improves multi-turn tool use, instruction following, and input grounding for fast, local voice/device assistants, interactive personas, and embedded/Edge interfaces. The model combines natural conversation with multi-turn agentic capabilities to power voice/device assistants, interactive characters and robotic/IoT interfaces running on GeForce RTX, Jetson, and RTX/DGX Spark. Experiments on 21 benchmarks show that Nemotron 3.5 Bolt E4B 4GB achieves the highest average accuracy while generating the fewest output tokens in both reasoning and non-reasoning modes, compared with Gemma-4-E4B-it and Qwen3.5-4B. The model is released with a DFlash drafter. With DFlash, Bolt E4B achieves $2.74\times$ the per-user decode speed of inference without speculative decoding at concurrency~1, and $1.21\times$ the generation throughput of Gemma-4-E4B-it with MTP at batch size 1.
BibTeX
@misc{nvidia2026nemotron,
  author = {NVIDIA and Adi Margolin and Aditya Malte and Aishwarya Chhabra and Alex Hodson and Ameya Sunil Mahabaleshwarkar and Ambrish Dantrey and Amy Shen and Andrii Skliar and Annamalai Chockalingam and Anusha Pant and Ashwath Aithal and Ashwin Kumar and Besmira Nushi and Bhavesh Pawar and Bryan Catanzaro and Bryan Yu and Chantal Hwang and Chenhan Yu and Chris Alexiuk and Conner Currie and Cynthia Gao and David Mosallanezhad and Dina Yared and Ehsan Hosseini Asl and Eileen Long and Erick Galinkin and Felix Friedmann and Giedrius Burachas and Gorkem Batmaz and Grzegorz Chlebus and Guillermo Siman and Haifeng Qian and Hao Guo and Hayley Ross and Hrishith Thadicherla and Hui Wang and Huiying Li and Ike Nnoli and Irina Filatova and Iris Liu and Ji-Eun Lee and Jian Leong and Jian Zhang and Jiaqi Zeng and Joey Conway and Jonathan Cohen and Jose Vergara and Julien Veron Vialard and Kari Briski and Katherine Luna and Kedar Pandurang Potdar and Kevin Ning and Khalil Ben Khaled and Krzysztof Pawelec and Laura Dykes and Leili Tavabi and Lily Lee and Linda Liu and Liwei Ding and Maka Dong and Manoj Kilaru and Maria Bagdasarova and Matthew Lee and Meredith Price and Micah Schaffer and Michael Boone and Michael Fukuyama and Michael Hasper and Nancy Agarwal and Negar Habibi and Oleksii Kuchaiev and Oluwatobi Olabiyi and Pablo Ribalta and Payal Motwani and Pinky Xu and Prasoon Varshney and Pritam Biswas and Puhui Meng and Rameshwar Shivbhakta and Ran Zilberstein and Reka Kovacs and Richard Mazzarese and Rishabh Garg and Rita Fernandes Neves and Rochelle Woods and Ruby Ye and Ruvaid Virk and Ryan Hill and Saeid Motiian and Seph Mard and Shai Ben Nun and Shaona Ghosh and Sharad Yadav and Sudeep Sabnis and Suhas Hariharapura Sheshadri and Terry Kong and Thanh Do and Thinh Le and Udi Karpas and Viet Anh Trinh and Vinay Raman and Vinita Sharma and Warren Drones and Wenwen Gao and Whei Hsueh and Will Efstratis and Yoshi Suhara and Yunsheng Liu and Zhenming Wang and Zijia Chen and Zuhair Ahmed},
  title = {{Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational Agents}},
  year = {2026},
  eprint = {2611.20771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20771},
}
Figure