NVIDIA’s Nemotron 3.5 Bolt E4B 4GB is a highly efficient small language model (SLM) optimized for fast, local conversational agents on Edge devices. By leveraging pure reinforcement learning (RL) post-training and speculative decoding, it achieves state-of-the-art accuracy in tool use and instruction following while maintaining a tiny 4GB memory footprint and delivering up to a 2.74× speedup in inference.
What Problem Does This Paper Address?
Building conversational agents for local deployment (e.g., voice assistants, robots, or IoT devices) presents a "trilemma" of constraints: intelligence, memory, and latency.
Intelligence: The model must reliably follow instructions and use tools across multiple turns.
Memory: It must fit within the limited VRAM of consumer GPUs or Edge modules like NVIDIA Jetson.
Latency: For natural interaction, the "Time to First Token" (TTFT) and generation speed must be near-instant.
Existing models often struggle to balance these, either being too large for local VRAM or too verbose and slow for real-time voice interfaces. Nemotron 3.5 Bolt aims to maximize "intelligence per GB" specifically for these fast-response scenarios.
Methods
The researchers started with Gemma-4-E4B-it as a base and applied several novel post-training and deployment strategies:
Pure RL Post-Training: Unlike the standard pipeline that uses Supervised Fine-Tuning (SFT) followed by RL, the authors found that SFT actually regressed conversational quality and increased verbosity. They opted for pure RL post-training using RLVR (Reinforcement Learning from Verifiable Rewards) and RLHF via the NeMo-RL framework.
Dual-Mode Reasoning: The model is trained to operate in both reasoning (enable_thinking=true) and non-reasoning (enable_thinking=false) modes. Every training sample was used in both modes to ensure high accuracy even when the "chain-of-thought" is disabled for speed.
Verbosity Control: To prevent the "chatty" nature of many LLMs, they implemented group-relative length penalties. This forces the model to be concise—a critical feature for voice-based agents.
DFlash Speculative Decoding: The model is released with a DFlash drafter (a 360M parameter Transformer). This drafter predicts multiple future tokens that the main model then verifies in parallel, significantly boosting throughput.
Quantization: The model is optimized for local deployment using Q4_K_M quantization with importance-matrix (imatrix) calibration, ensuring minimal accuracy loss at 4-bit precision.
Datasets
The model was post-trained on a diverse blend of environments (see Table 1 below) focusing on multi-turn tool use, instruction hierarchy, and structured outputs.
Environments and source datasets used in post-training
Evaluation spanned 21 benchmarks, including:
Conversational Proficiency: EQ-Bench v3, MultiChallenge, and RoleMRC.
Tool Use: BFCL-v3, ToolSandbox, and a new multi-turn extension called When2Call-MT.
Internal Brevity Benchmark: Used to measure the model's ability to provide high-quality, concise answers suitable for spoken conversation.
Key Insights and Ideas
Conciseness is a Feature, Not a Bug: In the "Smart and Fast" quadrant, Bolt E4B stands out by being significantly less verbose than competitors like Qwen3.5-4B while maintaining higher accuracy. For example, on reasoning tasks, it generates roughly 1/5th the tokens of Qwen while achieving better scores.
The SFT Regression: The discovery that SFT on diverse "teacher" data can hurt conversational quality for small models is a significant finding for researchers looking to optimize SLMs.
Multimodal Resilience: Despite being post-trained only on text, the model retains the vision and audio capabilities of its base model. This suggests that high-quality text RL doesn't necessarily "break" the underlying multimodal alignments.
Accuracy vs. Verbosity trade-off across benchmarks
Key Results
Performance: Bolt E4B achieved the highest average accuracy across 21 benchmarks compared to Gemma-4-E4B-it and Qwen3.5-4B.
Speed: With the DFlash drafter, the model achieves 2.74× the per-user decode speed at concurrency 1 on an RTX PRO 6000.
Efficiency: As shown in Figure 1, Bolt E4B occupies the "Smart and Fast" top-left corner, offering a superior balance of benchmark scores and TTFT compared to much larger models.
Benchmark score vs. Time to First Token (TTFT)
Live Testing: The model was successfully tested in real-time scenarios, including controlling a robot (Reachy) and acting as an NPC in Minecraft, demonstrating its ability to handle grounding and tool calls autonomously.
Limitations and Future Work
Vision Degradation: The authors noted a slight but consistent drop in vision performance (e.g., on MMMU-Pro Vision) compared to the base model, likely due to the text-only RL phase.
Audio Constraints: Evaluation on audio benchmarks was limited by a 30-second input window; many real-world conversations exceed this.
Language Support: While the model retains some multilingual capabilities, the post-training was English-only. Future work aims to restore and improve vision capabilities and expand multilingual support.
Summary updated
Abstract
We introduce Nemotron 3.5 Bolt E4B 4GB, an open and efficient small language model further post-trained from Gemma-4-E4B-it and optimized for speed and intelligence per GB of memory used for fast conversational agents. The model carries forward Gemma's strong conversational capabilities and further improves multi-turn tool use, instruction following, and input grounding for fast, local voice/device assistants, interactive personas, and embedded/Edge interfaces. The model combines natural conversation with multi-turn agentic capabilities to power voice/device assistants, interactive characters and robotic/IoT interfaces running on GeForce RTX, Jetson, and RTX/DGX Spark. Experiments on 21 benchmarks show that Nemotron 3.5 Bolt E4B 4GB achieves the highest average accuracy while generating the fewest output tokens in both reasoning and non-reasoning modes, compared with Gemma-4-E4B-it and Qwen3.5-4B. The model is released with a DFlash drafter. With DFlash, Bolt E4B achieves $2.74\times$ the per-user decode speed of inference without speculative decoding at concurrency~1, and $1.21\times$ the generation throughput of Gemma-4-E4B-it with MTP at batch size 1.
BibTeX
@misc{nvidia2026nemotron,
author = {NVIDIA and Adi Margolin and Aditya Malte and Aishwarya Chhabra and Alex Hodson and Ameya Sunil Mahabaleshwarkar and Ambrish Dantrey and Amy Shen and Andrii Skliar and Annamalai Chockalingam and Anusha Pant and Ashwath Aithal and Ashwin Kumar and Besmira Nushi and Bhavesh Pawar and Bryan Catanzaro and Bryan Yu and Chantal Hwang and Chenhan Yu and Chris Alexiuk and Conner Currie and Cynthia Gao and David Mosallanezhad and Dina Yared and Ehsan Hosseini Asl and Eileen Long and Erick Galinkin and Felix Friedmann and Giedrius Burachas and Gorkem Batmaz and Grzegorz Chlebus and Guillermo Siman and Haifeng Qian and Hao Guo and Hayley Ross and Hrishith Thadicherla and Hui Wang and Huiying Li and Ike Nnoli and Irina Filatova and Iris Liu and Ji-Eun Lee and Jian Leong and Jian Zhang and Jiaqi Zeng and Joey Conway and Jonathan Cohen and Jose Vergara and Julien Veron Vialard and Kari Briski and Katherine Luna and Kedar Pandurang Potdar and Kevin Ning and Khalil Ben Khaled and Krzysztof Pawelec and Laura Dykes and Leili Tavabi and Lily Lee and Linda Liu and Liwei Ding and Maka Dong and Manoj Kilaru and Maria Bagdasarova and Matthew Lee and Meredith Price and Micah Schaffer and Michael Boone and Michael Fukuyama and Michael Hasper and Nancy Agarwal and Negar Habibi and Oleksii Kuchaiev and Oluwatobi Olabiyi and Pablo Ribalta and Payal Motwani and Pinky Xu and Prasoon Varshney and Pritam Biswas and Puhui Meng and Rameshwar Shivbhakta and Ran Zilberstein and Reka Kovacs and Richard Mazzarese and Rishabh Garg and Rita Fernandes Neves and Rochelle Woods and Ruby Ye and Ruvaid Virk and Ryan Hill and Saeid Motiian and Seph Mard and Shai Ben Nun and Shaona Ghosh and Sharad Yadav and Sudeep Sabnis and Suhas Hariharapura Sheshadri and Terry Kong and Thanh Do and Thinh Le and Udi Karpas and Viet Anh Trinh and Vinay Raman and Vinita Sharma and Warren Drones and Wenwen Gao and Whei Hsueh and Will Efstratis and Yoshi Suhara and Yunsheng Liu and Zhenming Wang and Zijia Chen and Zuhair Ahmed},
title = {{Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational Agents}},
year = {2026},
eprint = {2611.20771},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.20771},
}