TL;DR
NariQ is a new post-training quantization (PTQ) method that enables 27B-parameter models to run at sub-3-bit precision without sacrificing their ability to handle complex, long-horizon agentic tasks. By calibrating specifically on model-generated tokens from real agent sessions, NariQ achieves full-precision performance at just 2.5 bits per weight, making high-capability agents accessible on consumer-grade GPUs.
What Problem Does This Paper Address?
As Large Language Models (LLMs) are increasingly deployed as "agents"—performing multi-step tasks like software engineering that require tool use and 100K+ token contexts—standard quantization methods are starting to show their cracks.
Most existing quantization techniques (like GPTQ) are calibrated on short snippets of web text and evaluated using perplexity. While a 3-bit model might still look "smart" on a short math quiz, it often fails miserably when asked to navigate a complex codebase over 150 tool calls. For example, previous 3-bit versions of Qwen-27B solved only about 63% as many tasks as the original BF16 model on the DeepSWE benchmark. The authors identify a fundamental mismatch: we are calibrating and judging models on data that doesn't reflect how agents actually work.
Methods
NariQ introduces a specialized "recipe" for post-training quantization that moves beyond simple weight-rounding:
- Agent-Aware Calibration: Instead of using generic web text, the authors calibrate the model using traces of actual agent sessions. Crucially, they use a masked loss that only counts the tokens the model writes (like reasoning and tool calls), ignoring the massive amounts of tool output and file content that typically dominate the context window.
- Trellis-Coded Quantization: They utilize trellis coding, a more sophisticated way to map weights to discrete bit values than standard scalar quantization. As shown in Figure 1, this switch alone significantly reduces the divergence from the original model.
- Sensitivity-Based Bit Allocation: Not all layers are equally important. NariQ uses a gradient-based approach to estimate the "loss" each module contributes and solves a knapsack problem to allocate more bits to the most sensitive layers while staying within a total bit budget.
- Banded Output Layer: The final layer of the model is divided into "bands" based on token frequency. Likely tokens get more precision, while rare tokens are protected with a minimum 2-bit floor to prevent the model from losing its general capabilities.
- Scale Tuning: Once weights are quantized, they fine-tune the scales and norms using a 70:30 mix of agent traces and general reasoning data (math, science, chat) to ensure the model remains a versatile "all-rounder."
Datasets
- Calibration: Agent sessions of the Qwen3.8-27B (BF16) model performing open-source software tasks.
- Agentic Benchmarks: DeepSWE v1.1 (software engineering tasks requiring ~150 tool calls) and Terminal Bench 2.1 (command-line tasks).
- General Benchmarks: AIME 2025+2026 (math), RealWorldQA (visual understanding), and MGSM (multilingual reasoning).
Key Insights and Ideas
- Perplexity is a Liar for Agents: A model can have great perplexity on Wikipedia but still be a terrible agent. The "aha moment" here is that an agent's quality depends specifically on the tokens it chooses to write, not how well it predicts the static context it's given.
- The "Tool Output" Noise: In long-horizon tasks, tool outputs (like compiler logs or file contents) can make up the vast majority of the context. If you calibrate on all tokens equally, the quantization process optimizes for predicting the tool's behavior rather than the model's own reasoning. Masking these out is essential.
- The 2-Bit Floor: The authors found that dropping any layer or output band to 1 bit caused catastrophic failures in general reasoning. Maintaining a 2-bit minimum is a critical safety rail for sub-3-bit models.
Key Results
NariQ represents a significant leap for low-bit models in agentic workflows:
- Matching Full Precision: At 3.0 and 2.5 bits per weight (bpw), NariQ-quantized models performed on par with the original BF16 version. On DeepSWE, the 3.0 bpw version actually solved more tasks (47 vs 44) than the original, likely due to the noise inherent in agentic evaluations.
- Outperforming Baselines: Compared to other popular quantization formats like GSQ-RCO or EXL3 at similar bitrates, NariQ consistently solved significantly more agentic tasks (see Table 1).
- General Ability Preservation: Despite the heavy focus on agentic calibration, the 3.0 bpw model stayed within 1 point of the original on math (AIME) and visual (RealWorldQA) benchmarks.
- Efficiency: The 2.5 bpw model fits into roughly 8.4 GB, allowing a powerful 27B model to run comfortably on a single consumer GPU while maintaining high inference speeds (up to 112 tokens/s at 4K context).
Limitations and Future Work
While NariQ pushes the boundaries of sub-3-bit quantization, some challenges remain:
- The 2.0 bpw Gap: At 2.0 bpw, performance begins to trail significantly (dropping 7.3 points on DeepSWE). The authors suggest that training beyond just scales and norms might be necessary to recover this loss.
- Unquantized Components: The multi-token prediction module is currently not quantized, and the KV cache—which grows massive in 100K+ token contexts—still needs to be addressed to fit these models into 16GB GPUs for long-horizon use.
- Hardware Optimization: The current inference implementation is highly optimized for specific NVIDIA GPU generations (SM120), with slower performance on older architectures.
Summary updated