TL;DR
MPQ (Message-Passing Quantization) rethinks post-training quantization as a probabilistic inference problem rather than a greedy search. By using message passing to maintain "soft" beliefs about weight assignments before committing to a final value, it significantly reduces error in ultra-low-bit (2-3 bit) large language model compression.
What Problem Does This Paper Address?
To run massive Large Language Models (LLMs) on consumer hardware, we must compress their weights—often from 16-bit floats down to just 2, 3, or 4 bits. This process, called Post-Training Quantization (PTQ), is tricky because changing one weight affects the optimal value for its neighbors.
Existing state-of-the-art solvers like GPTQ or LDLQ are "greedy." They process weight blocks sequentially, making an immediate "hard decision" on the best discrete codeword for a block and then trying to compensate for the resulting error in subsequent blocks. In the aggressive low-bit regime (e.g., 2 bits), there are very few codewords to choose from. A greedy choice that looks good locally might be globally disastrous, with no way for the algorithm to "backtrack" or reconsider.
Methods
The authors propose Message-Passing Quantization (MPQ), which treats PTQ as a posterior inference problem. Instead of picking one codeword immediately, the algorithm maintains a "soft" probability distribution over multiple likely candidates for every block.
The method consists of two main stages:
- Finite-Temperature VAMP: It uses Vector Approximate Message Passing (VAMP) to let blocks "talk" to each other. Instead of passing a single hard value, blocks pass "messages" representing their current beliefs about which codewords are likely. This allows the solver to explore uncertainty and find a better global basin before committing.
- Hard Refinement: Once the soft updates converge, the algorithm projects the distributions to the most likely hard codewords (Marginal MAP) and performs a final "polishing" step using Block Coordinate Descent (CD) to ensure local optimality.
This framework unifies existing solvers: the authors show that GPTQ and LDLQ are essentially "zero-temperature" versions of MPQ that discard all uncertainty information instantly.
Datasets
The researchers evaluated MPQ across several popular model families, including Llama (1B to 70B), Qwen3 (8B), and Mistral (7B).
- Calibration: 128 sequences from the C4 dataset.
- Evaluation: Perplexity was measured on WikiText-2 and C4.
- Downstream Tasks: Zero-shot accuracy was tested on a suite of benchmarks including ARC (Challenge & Easy), HellaSwag, PIQA, and Winogrande.
Key Insights and Ideas
- The "Early Commitment" Tax: Greedy solvers pay a heavy price in low-bit regimes because they commit to a single trajectory through the search space too early. MPQ's "soft" approach postpones this commitment, leading to better global solutions.
- Probabilistic Unification: By casting PTQ as inference, the authors provide a theoretical bridge between classical signal processing (message passing) and modern LLM compression.
- Synergy with Preprocessing: MPQ isn't a replacement for tricks like activation scaling (AWQ) or weight rotations (QuIP/QuRot); it's a better solver that composes favorably with them. The paper includes theoretical analysis showing how diagonal scaling (like AWQ) actually improves the "effective precision" that the message-passing stage sees.
Key Results
MPQ consistently outperforms strong baselines like GPTQ, QuantEase, and OJBKQ, with the most dramatic gains seen at the lowest bit-widths.
- 2-Bit Lattice Quantization: On Llama 3.2-1B, MPQ achieved a WikiText-2 perplexity of 31.67, a massive improvement over the LDLQ baseline of 42.80.
- 3-Bit Scalar Quantization: Across the Llama family, MPQ reduced perplexity by significant margins (e.g., from 14.26 to 13.65 on Llama 3.2-1B compared to the strongest randomized baseline).
- Zero-Shot Accuracy: On Qwen3-8B at 2-bit, MPQ improved average zero-shot accuracy by 1.9% over the next best baseline.
Limitations and Future Work
- Computational Overhead: MPQ is slower than simple greedy methods. While it's still an "offline" process (done once per model), the extra VAMP iterations and CD passes add to the quantization time. However, the authors show that for scalar quantization, even a few VAMP iterations without CD can beat existing baselines.
- Codebook Specificity: The current implementation focuses on uniform scalar and specific lattice (E8P12, D4) codebooks. Adapting it to learned vector codebooks (like AQLM) or trellis-coded quantization (like QTIP) would require extending the discrete denoiser logic.
- Theoretical Gap: The theoretical analysis relies on "Right Orthogonally Invariant" (ROI) surrogates. While these provide excellent qualitative guidance, a tight characterization for real-world, non-ROI weight matrices remains an open challenge.
The code is available at https://github.com/poseidonchan/MPQ.
Summary updated