TL;DR
JointQ is a drop-in replacement for the industry-standard GPTQ solver that jointly optimizes quantization scales and discrete integer assignments. By coupling these two components through a Hessian-aware local search and Tikhonov regularization, it significantly preserves model accuracy in challenging 3-bit and per-channel regimes where standard methods often collapse.
What Problem Does This Paper Address?
Large Language Models (LLMs) are massive, making post-training quantization (PTQ)—compressing 16-bit weights into 4-bit or 3-bit integers—essential for deployment. The current "de facto" standard for this is GPTQ.
However, GPTQ has a structural bottleneck: it decouples the optimization of scales (which define the quantization grid's spacing) from integers (which select the points on that grid). GPTQ typically picks a scale using a simple heuristic and then greedily assigns integers. While this works fine for 4-bit quantization with small groups, it fails in "aggressive" regimes. In 3-bit or per-channel quantization, the grid is so sparse that picking the wrong scale makes it impossible for the integers to accurately reconstruct the original weights, leading to catastrophic drops in model performance.
Methods
JointQ moves beyond the one-pass greedy approach of GPTQ by treating scales and integers as coupled variables. Its methodology rests on three pillars:
- Scale Elimination & Joint Search: The authors show that for any given set of integers, the optimal scales can be solved in closed form using the Hessian (second-order activation statistics). JointQ uses a local search where each "move" changes a single integer and immediately recomputes the optimal scale for that group. Periodically, it re-solves all scales for an entire row jointly to account for cross-group coupling.
- Multi-Candidate Warm Start: To ensure it never performs worse than the baseline, JointQ starts by evaluating multiple candidates—including the standard GPTQ solution and a "Clip-Optimize" solution. It selects the best starting point for each output row independently based on calibration error.
- Diagonal Tikhonov Regularization: The Hessian matrix can be ill-conditioned when calibration data is scarce. JointQ introduces a diagonal regularizer that penalizes quantization errors more heavily on "high-energy" input features. This stabilization is critical for preventing the model from "breaking" in 3-bit settings.
Importantly, JointQ is a drop-in replacement. It produces the exact same output format as GPTQ, meaning it works with all existing optimized inference kernels and checkpoints.
Datasets
The authors evaluated JointQ across four major model families: Llama-2-7B, Llama-3-8B, Qwen3-8B, and Qwen3-14B.
- Calibration: 1024 sequences from the C4 dataset.
- Perplexity Evaluation: WikiText-2.
- Zero-Shot Accuracy: A battery of six tasks including ARC-Challenge, ARC-Easy, PIQA, WinoGrande, HellaSwag, and BoolQ.
Key Insights and Ideas
- The "Aha" Moment on Coupling: The paper demonstrates that in low-bit regimes, the "direction gap" (how well integers align with weights) and the "scale gap" (how well the grid fits the weight distribution) are inseparable. Optimizing them jointly allows the solver to find "sweet spots" that greedy solvers miss.
- Regularization is the Secret Sauce: An ablation study revealed that the local search alone isn't enough to save 3-bit per-channel quantization. The Tikhonov regularization is what actually mitigates the severe perplexity degradation, suggesting that "robustness" is just as important as "optimization" in PTQ.
- Guaranteed Non-Degradation: Because the GPTQ solution is included in the initial candidate pool, unregularized JointQ is mathematically guaranteed to have a layer-wise calibration error no higher than GPTQ. It essentially "refines" GPTQ rather than replacing it blindly.
Key Results
JointQ consistently outperformed vanilla GPTQ across various configurations:
- Accuracy Gains: Improved the six-task average zero-shot accuracy in 14 out of 16 model/setting comparisons.
- Low-Bit Resilience: The benefits were most dramatic in the hardest settings. In 3-bit per-channel quantization, the mean accuracy gain over GPTQ was a massive 0.1559.
- Saving "Broken" Models: For Llama-3-8B in the 3-bit per-channel setting, GPTQ's perplexity exploded to ~3381 (essentially unusable), while JointQ brought it down to a functional 13.09.
- Robustness: JointQ showed much lower sensitivity to the specific random seed used for calibration data compared to GPTQ.
Limitations and Future Work
- Quantization Time: JointQ is significantly more computationally expensive than GPTQ during the offline quantization phase (ranging from 2.6x to 15.7x slower). While this doesn't affect inference speed, it increases the "time-to-deploy."
- Hyperparameters: The regularization strength ($\lambda$) was manually tuned. A more automated, data-driven way to set these values per layer would be a valuable extension.
- Zero-Point Optimization: While the paper discusses asymmetric quantization in the appendix, the current local search does not jointly optimize "zero-points" (the offset of the quantization grid), which remains a potential area for further improvement.
- Model Dependence: While accuracy generally improved, the effect on language-modeling perplexity was occasionally model-dependent, particularly on the Qwen families.
Code Availability: The implementation is available in the OneComp library.
Summary updated