JointQ: Joint Scale-and-Integer Quantization for Large Language Models

Keiji Kimura, Yuma IchikawaarXiv 2026 · arXiv

TL;DR

JointQ is a drop-in replacement for the industry-standard GPTQ solver that jointly optimizes quantization scales and discrete integer assignments. By coupling these two components through a Hessian-aware local search and Tikhonov regularization, it significantly preserves model accuracy in challenging 3-bit and per-channel regimes where standard methods often collapse.

What Problem Does This Paper Address?

Large Language Models (LLMs) are massive, making post-training quantization (PTQ)—compressing 16-bit weights into 4-bit or 3-bit integers—essential for deployment. The current "de facto" standard for this is GPTQ.

However, GPTQ has a structural bottleneck: it decouples the optimization of scales (which define the quantization grid's spacing) from integers (which select the points on that grid). GPTQ typically picks a scale using a simple heuristic and then greedily assigns integers. While this works fine for 4-bit quantization with small groups, it fails in "aggressive" regimes. In 3-bit or per-channel quantization, the grid is so sparse that picking the wrong scale makes it impossible for the integers to accurately reconstruct the original weights, leading to catastrophic drops in model performance.

Methods

JointQ moves beyond the one-pass greedy approach of GPTQ by treating scales and integers as coupled variables. Its methodology rests on three pillars:

  1. Scale Elimination & Joint Search: The authors show that for any given set of integers, the optimal scales can be solved in closed form using the Hessian (second-order activation statistics). JointQ uses a local search where each "move" changes a single integer and immediately recomputes the optimal scale for that group. Periodically, it re-solves all scales for an entire row jointly to account for cross-group coupling.
  2. Multi-Candidate Warm Start: To ensure it never performs worse than the baseline, JointQ starts by evaluating multiple candidates—including the standard GPTQ solution and a "Clip-Optimize" solution. It selects the best starting point for each output row independently based on calibration error.
  3. Diagonal Tikhonov Regularization: The Hessian matrix can be ill-conditioned when calibration data is scarce. JointQ introduces a diagonal regularizer that penalizes quantization errors more heavily on "high-energy" input features. This stabilization is critical for preventing the model from "breaking" in 3-bit settings.

Importantly, JointQ is a drop-in replacement. It produces the exact same output format as GPTQ, meaning it works with all existing optimized inference kernels and checkpoints.

Datasets

The authors evaluated JointQ across four major model families: Llama-2-7B, Llama-3-8B, Qwen3-8B, and Qwen3-14B.

  • Calibration: 1024 sequences from the C4 dataset.
  • Perplexity Evaluation: WikiText-2.
  • Zero-Shot Accuracy: A battery of six tasks including ARC-Challenge, ARC-Easy, PIQA, WinoGrande, HellaSwag, and BoolQ.

Key Insights and Ideas

  • The "Aha" Moment on Coupling: The paper demonstrates that in low-bit regimes, the "direction gap" (how well integers align with weights) and the "scale gap" (how well the grid fits the weight distribution) are inseparable. Optimizing them jointly allows the solver to find "sweet spots" that greedy solvers miss.
  • Regularization is the Secret Sauce: An ablation study revealed that the local search alone isn't enough to save 3-bit per-channel quantization. The Tikhonov regularization is what actually mitigates the severe perplexity degradation, suggesting that "robustness" is just as important as "optimization" in PTQ.
  • Guaranteed Non-Degradation: Because the GPTQ solution is included in the initial candidate pool, unregularized JointQ is mathematically guaranteed to have a layer-wise calibration error no higher than GPTQ. It essentially "refines" GPTQ rather than replacing it blindly.

Key Results

JointQ consistently outperformed vanilla GPTQ across various configurations:

  • Accuracy Gains: Improved the six-task average zero-shot accuracy in 14 out of 16 model/setting comparisons.
  • Low-Bit Resilience: The benefits were most dramatic in the hardest settings. In 3-bit per-channel quantization, the mean accuracy gain over GPTQ was a massive 0.1559.
  • Saving "Broken" Models: For Llama-3-8B in the 3-bit per-channel setting, GPTQ's perplexity exploded to ~3381 (essentially unusable), while JointQ brought it down to a functional 13.09.
  • Robustness: JointQ showed much lower sensitivity to the specific random seed used for calibration data compared to GPTQ.

Limitations and Future Work

  • Quantization Time: JointQ is significantly more computationally expensive than GPTQ during the offline quantization phase (ranging from 2.6x to 15.7x slower). While this doesn't affect inference speed, it increases the "time-to-deploy."
  • Hyperparameters: The regularization strength ($\lambda$) was manually tuned. A more automated, data-driven way to set these values per layer would be a valuable extension.
  • Zero-Point Optimization: While the paper discusses asymmetric quantization in the appendix, the current local search does not jointly optimize "zero-points" (the offset of the quantization grid), which remains a potential area for further improvement.
  • Model Dependence: While accuracy generally improved, the effect on language-modeling perplexity was occasionally model-dependent, particularly on the Qwen families.

Code Availability: The implementation is available in the OneComp library.

Summary updated

Abstract

GPTQ is the de facto standard solver for layer-wise post-training quantization (PTQ) of large language models, yet the solver itself has received less attention than weight preprocessing and cross-layer objectives. We target a bottleneck of GPTQ: its greedy procedure optimizes continuous group scales separately from the assignment of discrete integers. Evaluating each integer candidate with its closed-form optimal scale has been used for per-channel quantization; JointQ extends it to group-wise quantization, where a row's scales are coupled through the Hessian, and applies it to refine the GPTQ solution. Each local-search move changes one integer and recomputes its group's optimal scale in closed form, accounting for the other groups through the full Hessian; all scales are periodically re-solved jointly. The search starts from a row-wise selection among scale-refitted candidates including the GPTQ solution, so unregularized JointQ never has a larger layer-wise calibration error than GPTQ. JointQ preserves the GPTQ output format and is thus a drop-in replacement. Across four models and four settings, JointQ improves six-task average zero-shot accuracy over vanilla GPTQ in 14 of 16 comparisons and perplexity in 12 of 16. The mean accuracy gain increases from 0.0044 in W4G128 to 0.1559 in W3 per-channel, and JointQ also improves average accuracy over GPTQ with MSE-based scale search and activation ordering in all eight per-channel comparisons, though rarely in G128. An ablation shows that JointQ's diagonal Tikhonov regularization mitigates severe W3 per-channel perplexity degradation, so the largest gains are not due to the local search alone. Overall, JointQ's primary empirical benefit is downstream accuracy preservation in challenging low-bit and large-group regimes. Code is available in the OneComp library at https://github.com/FujitsuResearch/OneCompression.
BibTeX
@misc{kimura2026jointq,
  author = {Keiji Kimura and Yuma Ichikawa},
  title = {{JointQ: Joint Scale-and-Integer Quantization for Large Language Models}},
  year = {2026},
  eprint = {2611.20384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20384},
}
Figure