TL;DR
BARQ (Balanced Assignment Refinement for Quantization) is a novel post-training refinement method that significantly improves the quality of low-bit Large Language Model (LLM) quantization. By using optimal transport to ensure every codeword in a codebook is effectively utilized during the fitting process, BARQ achieves better model performance (lower perplexity and higher accuracy) without changing the standard inference path.
What Problem Does This Paper Address?
As LLMs scale to hundreds of billions of parameters, the memory bandwidth required to move weights from storage to the processor becomes a massive bottleneck. Codebook-based quantization (a form of Vector Quantization) is a promising solution, representing groups of weights with shared "codewords" to reach extremely low bitrates (e.g., 2 bits per parameter).
However, these methods suffer from a "dead codeword" problem. During the fitting process, standard algorithms often assign most weight blocks to just a few popular codewords, leaving others unused or poorly updated. This imbalance wastes the limited representational capacity of the codebook.
Methods
BARQ introduces a refinement stage for existing codebooks that forces a "balanced" assignment during training. The pipeline consists of four main steps:
- Curvature-Aware Cost Calculation: It uses Hessian-based curvature information (standard in methods like GPTQ) to determine which weights are most sensitive to quantization error.
- Balanced-Entropic E-step: Instead of a simple "nearest-neighbor" assignment, BARQ treats assignment as an Optimal Transport problem. It uses the Sinkhorn algorithm with uniform marginal constraints to compute a "soft" assignment. This forces every codeword to receive an equal share (1/K) of the total fitting mass.
- Barycentric M-step: Codewords are updated to the weighted average (barycenter) of the blocks assigned to them. The authors prove that this update minimizes the fitting objective for the given soft assignments.
- Hard Deployment: Crucially, once the codebook is refined, the soft assignments are discarded. For actual inference, the model uses standard hard nearest-codeword lookups. This means BARQ improves the quality of the codebook without requiring specialized hardware or changing the inference logic.
Datasets
The authors evaluated BARQ across a diverse set of LLMs, including the Llama 3 family (1B and 8B) and the Qwen3 family (ranging from 1.7B to 32B).
Evaluation benchmarks included:
- WikiText-2: Used to measure language modeling perplexity (PPL).
- Zero-shot Accuracy Suite: A collection of five tasks including ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and WinoGrande.
Key Insights and Ideas
- Resuscitating Dead Codewords: The core "aha" moment is that forcing balance during refinement leads to a more globally optimal codebook, even if the final deployment returns to an imbalanced hard assignment.
- Theoretical Guarantee: The paper provides a rigorous proof that their balanced refinement reduces the fitting objective and establishes sufficient conditions for this to translate into lower distortion in the final "hard" quantized model.
- Inference Compatibility: Because BARQ only changes how the codebook values are calculated—not the indexing or lookup mechanism—it is a "drop-in" improvement for existing codebook-based quantization frameworks like GPT-VQ.
Key Results
Across 60 different configurations, BARQ consistently outperformed the state-of-the-art GPT-VQ baseline:
- Perplexity: Achieved a 7.0% average relative reduction in perplexity.
- Accuracy: Improved mean zero-shot accuracy by 3.0% across the five-task suite.
- Baseline Comparison: BARQ outperformed a wide range of external baselines, including GPTQ, HQQ, AQLM, and QuIP, at comparable bit budgets.
As shown in the baseline comparison table below, BARQ (Ours) achieves the best results (bolded) across all tested model scales.
Limitations and Future Work
- Low-Bit Focus: The study primarily focuses on extremely low bit widths (around 2 bits per value). The relative gains might diminish at higher bitrates where codebook capacity is less constrained.
- Sensitivity to Calibration: Like most post-training quantization (PTQ) methods, the results depend on the quality and representativeness of the calibration data.
- Future Directions: The authors suggest exploring BARQ across a broader range of architectures and investigating how this balanced refinement interacts with other techniques like weight rotation or sparse-dense decomposition.
Summary updated