Abstract
Layer-Projected Coordinate Descent (LPCD) is a unified framework for post-training quantization (PTQ) that shifts the optimization focus from individual layers to entire submodules, such as Attention or MLP blocks. By alternating between a continuous relaxation step and a projection step using standard layer-wise quantizers, LPCD effectively mitigates the error accumulation that typically plagues ultra-low-bit (2-bit and 3-bit) Large Language Models (LLMs). This framework provides a theoretical bridge between existing methods and achieves state-of-the-art performance while remaining compatible with hardware-efficient inference pipelines.
Introduction
Quantizing Large Language Models (LLMs)—reducing weight precision from 16-bit to 4-bit or lower—is essential for deploying these models on consumer-grade hardware. While popular methods like GPTQ and AWQ have made 4-bit quantization standard, they often struggle at ultra-low bit-widths (2-bit or 3-bit). LPCD addresses this by moving beyond "layer-wise" optimization to "submodule-wise" coordination, allowing layers to compensate for each other's quantization errors.
The Challenge of Low-Bit Quantization
Most post-training quantization methods optimize each linear layer in isolation. However, layers in modern Transformer architectures are deeply interconnected via residual connections and normalization layers (like RMSNorm). When quantized individually, small errors in early layers accumulate, leading to a "death by a thousand cuts" that significantly degrades model performance at 2-bit and 3-bit levels. Previous attempts to solve this, such as QEP and LoaQ, were specialized solutions; LPCD offers a unified theory that generalizes these approaches into a single optimization framework.
The LPCD Framework
LPCD treats quantization as an optimization problem across a submodule—a group of related layers that perform a specific function. The framework operates through an iterative two-step process:
- The Relaxation Step: The algorithm temporarily treats weights as continuous values, ignoring the discrete constraints of the quantization grid. It uses optimization techniques (such as Adam or closed-form least squares) to find the "ideal" continuous weights that minimize the error at the submodule's output.
- The Projection Step: These ideal continuous values are then mapped back onto the allowed quantized grid. Crucially, LPCD reuses existing, highly optimized layer-wise quantizers (like GPTQ or RTN) for this projection, ensuring the final model remains compatible with standard hardware kernels.
By alternating these steps, layers within a submodule can "cooperate." For example, an error introduced in the Up projection of an MLP can be partially offset by the optimization of the Down projection.
Experimental Setup
The researchers evaluated LPCD across several model families, including LLaMA 2, LLaMA 3, and Qwen, with parameter counts ranging from 7B to 14B. Calibration was performed using 128 samples from the C4 dataset. Performance was measured using WikiText-2 perplexity and zero-shot accuracy on benchmarks like ARC-Easy and PIQA.
Key Results and Discussion
LPCD demonstrates significant improvements in stability and accuracy, particularly at ultra-low bit-widths.
- 2-Bit Stability: While traditional methods often cause models to diverge (become incoherent) at 2-bit quantization, LPCD maintains stability. For instance, on LLaMA 3 8B at 2-bit, LPCD achieved a usable perplexity of ~87.5, whereas competing methods often failed to produce a functional model.
- Error Mitigation: The framework successfully suppresses the exponential growth of Mean Squared Error (MSE) across model layers. As shown in the comparative analysis, LPCD maintains a much lower error profile than QEP or LoaQ as the layer index increases.
- Orthogonality and Universality: LPCD is "quantizer-agnostic," meaning it can enhance simple Round-To-Nearest (RTN) schemes or more complex GPTQ pipelines. Furthermore, the framework is not limited to weights; it can be extended to quantize activations, KV-caches, and rotation matrices.
Conclusion
LPCD provides a robust, unified path for pushing LLM quantization to its limits. By optimizing at the submodule level, it recovers much of the accuracy lost during aggressive compression without the prohibitive cost of global fine-tuning. Future work may involve directly optimizing through non-linear components like RMSNorm and scaling the framework to frontier models exceeding 70B parameters.
Summary updated