BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization

Chenhang Cui, Xu Xie, Linrui Xu, Xiaohao Liu, Xingyu Zhu, Fei Shen, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

BARQ (Balanced Assignment Refinement for Quantization) is a novel post-training refinement method that significantly improves the quality of low-bit Large Language Model (LLM) quantization. By using optimal transport to ensure every codeword in a codebook is effectively utilized during the fitting process, BARQ achieves better model performance (lower perplexity and higher accuracy) without changing the standard inference path.

What Problem Does This Paper Address?

As LLMs scale to hundreds of billions of parameters, the memory bandwidth required to move weights from storage to the processor becomes a massive bottleneck. Codebook-based quantization (a form of Vector Quantization) is a promising solution, representing groups of weights with shared "codewords" to reach extremely low bitrates (e.g., 2 bits per parameter).

However, these methods suffer from a "dead codeword" problem. During the fitting process, standard algorithms often assign most weight blocks to just a few popular codewords, leaving others unused or poorly updated. This imbalance wastes the limited representational capacity of the codebook.

Figure 1: Comparison of Codebook Fitting. Prior methods leave many codewords unused, while BARQ ensures all are utilized.

Methods

BARQ introduces a refinement stage for existing codebooks that forces a "balanced" assignment during training. The pipeline consists of four main steps:

  1. Curvature-Aware Cost Calculation: It uses Hessian-based curvature information (standard in methods like GPTQ) to determine which weights are most sensitive to quantization error.
  2. Balanced-Entropic E-step: Instead of a simple "nearest-neighbor" assignment, BARQ treats assignment as an Optimal Transport problem. It uses the Sinkhorn algorithm with uniform marginal constraints to compute a "soft" assignment. This forces every codeword to receive an equal share (1/K) of the total fitting mass.
  3. Barycentric M-step: Codewords are updated to the weighted average (barycenter) of the blocks assigned to them. The authors prove that this update minimizes the fitting objective for the given soft assignments.
  4. Hard Deployment: Crucially, once the codebook is refined, the soft assignments are discarded. For actual inference, the model uses standard hard nearest-codeword lookups. This means BARQ improves the quality of the codebook without requiring specialized hardware or changing the inference logic.
Figure 2: Overview of the BARQ pipeline (VQ: vector quantization; MLP: multilayer perceptron). Curvature-Aware Cost: compute the distances between weight blocks and initial codewords, and set the groupwise entropic temperature. Balanced-Entropic E-step: use log-domain Sinkhorn scaling to obtain soft assignments with uniform codeword and block marginals. Barycentric M-step: update each codeword to the assignment-weighted barycenter of the blocks. Hard Deployment: discard the transport state and encode the weights with the refined codebook using the original nearest-codeword rule. Only the refined codebook and hard indices are stored.

Datasets

The authors evaluated BARQ across a diverse set of LLMs, including the Llama 3 family (1B and 8B) and the Qwen3 family (ranging from 1.7B to 32B).

Evaluation benchmarks included:

  • WikiText-2: Used to measure language modeling perplexity (PPL).
  • Zero-shot Accuracy Suite: A collection of five tasks including ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and WinoGrande.

Key Insights and Ideas

  • Resuscitating Dead Codewords: The core "aha" moment is that forcing balance during refinement leads to a more globally optimal codebook, even if the final deployment returns to an imbalanced hard assignment.
  • Theoretical Guarantee: The paper provides a rigorous proof that their balanced refinement reduces the fitting objective and establishes sufficient conditions for this to translate into lower distortion in the final "hard" quantized model.
  • Inference Compatibility: Because BARQ only changes how the codebook values are calculated—not the indexing or lookup mechanism—it is a "drop-in" improvement for existing codebook-based quantization frameworks like GPT-VQ.

Key Results

Across 60 different configurations, BARQ consistently outperformed the state-of-the-art GPT-VQ baseline:

  • Perplexity: Achieved a 7.0% average relative reduction in perplexity.
  • Accuracy: Improved mean zero-shot accuracy by 3.0% across the five-task suite.
  • Baseline Comparison: BARQ outperformed a wide range of external baselines, including GPTQ, HQQ, AQLM, and QuIP, at comparable bit budgets.
Table 1: Paired comparison with GPT-VQ across 60 configurations. BARQ lowers PPL in 54/60 config- urations and raises Avg. in 53/60; both improve in 47/60. Results cover six models in two panels. PPL is WikiText-2 perplexity; Avg. is the five-task zero-shot mean (%). For each metric, the GPT-VQ baseline and BARQ (Ours) are shown side by side and the better value is bold (superscript: Ours −GPT-VQ). Full per-task accuracies are in Appendix B.

As shown in the baseline comparison table below, BARQ (Ours) achieves the best results (bolded) across all tested model scales.

Table 2: Comparison against external baselines. BARQ achieves the lowest perplexity and highest accuracy on all six models.

Limitations and Future Work

  • Low-Bit Focus: The study primarily focuses on extremely low bit widths (around 2 bits per value). The relative gains might diminish at higher bitrates where codebook capacity is less constrained.
  • Sensitivity to Calibration: Like most post-training quantization (PTQ) methods, the results depend on the quality and representativeness of the calibration data.
  • Future Directions: The authors suggest exploring BARQ across a broader range of architectures and investigating how this balanced refinement interacts with other techniques like weight rotation or sparse-dense decomposition.

Summary updated

Abstract

As large language models (LLMs) grow in parameter count, model storage and parameter memory traffic have become major bottlenecks to efficient deployment. Codebook-based weight quantization reduces these costs, but imbalanced nearest-codeword assignments during fitting can leave some codewords insufficiently updated, limiting effective codebook utilization. To address this limitation, we propose Balanced Assignment Refinement for Quantization (BARQ), which improves quantization quality through balanced fitting of existing codebooks. Specifically, we first compute joint soft assignments between weight blocks and codewords through entropically regularized optimal transport with uniform marginals and curvature-weighted reconstruction costs, ensuring equal positive fitting mass for every codeword in the exact solution. We then refine the codewords through an assignment-weighted barycentric update, which we prove minimizes the fitting objective for fixed assignments. For finite Sinkhorn iterations, the implemented update retains this optimality provided all codeword masses exceed the denominator floor. Finally, we discard the soft assignments and use the refined codebook for standard hard nearest-codeword encoding, with our analysis establishing sufficient conditions for reducing hard-quantization distortion and evaluation loss. Across multiple LLMs, BARQ achieves lower perplexity and higher mean zero-shot accuracy than the evaluated baselines at comparable bit budgets. The code is available at https://github.com/chenhangcuisg-code/BARQ.
BibTeX
@misc{cui2026barq,
  author = {Chenhang Cui and Xu Xie and Linrui Xu and Xiaohao Liu and Xingyu Zhu and Fei Shen and Tat-Seng Chua},
  title = {{BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization}},
  year = {2026},
  eprint = {2610.04490},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.04490},
}
Figure