Quantizing Large Language Models (LLMs)—the process of shrinking their weights from 16-bit to 4-bit or even 3-bit—is essential for making them run on consumer hardware or reducing data center costs. However, finding the right "recipe" of quantization techniques is often considered a black art. A method that works perfectly for a Llama model might fail miserably on a Qwen model or at a different bit-precision.
In a new paper, researchers from MIT and NVIDIA introduce QDA (Closed-Loop LLM Quantization Design with Agents), a system that uses an LLM agent to automatically design and refine quantization recipes tailored to specific models and hardware.
TL;DR
QDA automates the design of quantization recipes by using an LLM agent to search through a library of techniques, guided by a fast KL-divergence-based evaluator. It consistently outperforms the best hand-designed public recipes, recovering up to 98.3% of original accuracy on massive 320B-parameter models with zero inference overhead.
What Problem Does This Paper Address?
Quantization isn't just one algorithm; it’s a "recipe" involving transformations (like rotations), rounding methods (like GPTQ), and calibration data. The challenge is threefold:
- Recipe Sensitivity: The best recipe changes depending on the model architecture and the target precision. There is no "one size fits all."
- Evaluation Cost: Modern "reasoning" models (like DeepSeek-R1) generate long chains of thought. A single full evaluation can take nearly 90 GPU-hours, making iterative search impossible.
- Engineering Friction: Most quantization tools are siloed. Mixing a technique from one codebase with another often requires manual, error-prone porting.
Methods
QDA treats quantization design as a closed-loop optimization problem managed by an LLM agent (specifically Claude 3.5 Opus). The system consists of three main pillars:
- The Agentic Loop: The agent proposes a recipe, the system builds it, and the evaluator scores it. The agent then reads the logs, diagnoses failures (like repetition loops), and adjusts the recipe for the next round.
- Fast Proxy Evaluation: To avoid the 90-hour evaluation bottleneck, QDA scores candidates using KL divergence. It measures how much the quantized model's token predictions differ from the original BF16 model on a set of cached "reference traces." This process takes minutes rather than hours and is highly correlated with final accuracy.
- Composable Library: The authors implemented 21 different Post-Training Quantization (PTQ) methods (including GPTQ, AWQ, and various rotations) behind a unified interface. This allows the agent to "stack" methods—for example, applying a Hadamard rotation followed by GPTQ and then scale distillation.
Datasets
The researchers evaluated QDA across four reasoning models: Qwen3.5-9B, Qwen3.6-35B-A3B (MoE), Qwen3.8-27B, and DeepSeek-R1-Distill-Llama-8B.
Evaluation benchmarks included:
- Math: AIME 2025 and HMMT.
- Science: GPQA-Diamond.
- Code: LiveCodeBench v6.
- Instruction Following: IFBench and IFEval.
Calibration was performed on a mix of standard datasets (UltraChat) and "self-generated" data where the model reasons on its own.
Key Insights and Ideas
- KL Divergence is a Superior Signal: The authors found that KL divergence on reference traces agrees with final accuracy 82% of the time, compared to only 73% for standard layer-wise error metrics. This makes it a reliable "compass" for the agent.
- The Agent as a Diagnostician: KL divergence can miss "catastrophic" failures like infinite repetition loops. QDA includes a "generation check" step where the agent runs full answers for promising candidates. If it sees a model looping, it can specifically add "scale distillation" or change calibration data to fix it.
- No Free Lunch in Recipes: Figure 2 highlights how the "best" public method swaps places depending on the model. For instance, AWQ beats GPTQ on Qwen3.5 at 4-bit, but loses to it at 3-bit. QDA's agent successfully navigates these flips.
Key Results
- Accuracy Gains: Across four models, QDA improved average accuracy over the strongest public recipes by 0.9, 1.4, and 2.2 percentage points at W4A16, W4A8, and W3A16 precisions, respectively.
- Scaling to 320B Parameters: On the massive GLM-5.3-Flash model using the new NVFP4 format, QDA recovered 98.3% of the original FP8 model's accuracy, outperforming official vendor-released checkpoints from NVIDIA and Red Hat AI.
- Efficiency: A full search costs only 64 GPU-hours and roughly $53 in agent tokens. Because QDA uses standard formats, the resulting models run on existing high-performance kernels (like Marlin or CUTLASS) with zero inference overhead.
Limitations and Future Work
While QDA significantly reduces human effort, it isn't entirely "hands-off" for brand-new architectures. A small amount of human-written code (an "adapter") is still needed to help the library understand the connectivity of a completely novel model type.
Future work could involve allowing the agent to design the kernels themselves or expanding the search space to include mixed-precision quantization, where different layers use different bit-widths.
Paper Title: QDA: Closed-Loop LLM Quantization Design with Agents
Authors: Xingyang Li, Tianlang Zhao, Song Han, Ligeng Zhu (MIT & NVIDIA)
Summary updated