Introduction
Quantization is a cornerstone of modern Large Language Model (LLM) deployment, enabling the compression of 32-bit floating-point weights into 4-bit or 2-bit integers to fit on consumer-grade hardware. However, this process inevitably creates a "quantization gap"—the numerical difference between the original weight ($W$) and its quantized counterpart ($W_q$). Traditionally, this residual error ($D = W - W_q$) is treated as useless noise and discarded.
This paper introduces Discarded Weight Recycling (DWR), a novel framework that treats these discarded values not as trash, but as "treasures" containing vital structural information. By mining and reintegrating these components, DWR significantly narrows the quantization gap with zero additional inference overhead.
Methodology
The DWR framework operates through a two-stage process designed to refine quantized weights without altering the model's final architecture or computational requirements.
1. Search Space Generation
Rather than simply adding the raw error back—which would violate the target quantization format—DWR generates a search space of candidate weights using low-rank decomposition. The most advanced variant, Activation-aware Feature Map (AFM) generation, utilizes Principal Component Analysis (PCA) to identify error components that most significantly impact the model's output activations. This ensures that the "recycled" information is functionally relevant to the model's performance.
2. The DWR Algorithm
DWR employs a greedy, layer-by-layer optimization strategy. For each layer, the framework evaluates various recycled weight candidates from the search space. Unlike standard Post-Training Quantization (PTQ) methods that optimize for local Mean Squared Error (MSE), DWR uses Global Perplexity (PPL) on a small calibration set as its selection criterion. This global perspective prevents the accumulation of errors across layers, often referred to as the "butterfly effect" in deep networks.
Crucially, the recycled components are designed to be divisible by existing quantization scales, allowing them to be merged directly into the integer weights. The resulting model remains a standard quantized model, requiring no new parameters and maintaining original inference speeds.
Experimental Results
The researchers evaluated DWR across several model families, including BLOOM-7B1, LLaMA2 (7B, 13B, 70B), and LLaMA3-8B, using a wide array of benchmarks such as WikiText2, C4, and various commonsense reasoning tasks.
As shown in the results, DWR consistently outperforms baseline methods like GPTQ and OmniQuant. For instance, on the LLaMA3-8B (INT4) model, adding DWR reduced WikiText2 perplexity from 9.96 to 6.26, nearly reaching the full-precision (FP16) baseline of 6.24. In ultra-low-bit scenarios (INT3), LLaMA2-7B saw its average reasoning accuracy jump from 59.12% to 62.21%, effectively recovering nearly all performance lost during quantization.
MMLU Performance
The impact of DWR is also evident in the Massive Multitask Language Understanding (MMLU) benchmark, which tests a model's knowledge across 57 subjects.
DWR provides a consistent lift in MMLU scores. For LLaMA2-13B at INT3, DWR improved the 0-shot average from 47.16% to 50.34%. Even at the extreme constraint of INT2, DWR managed to extract marginal gains, demonstrating the robustness of the recycling approach.
Discussion and Conclusion
The success of DWR highlights a critical insight: quantization error is not random noise but structured information. By prioritizing global perplexity over local MSE, DWR ensures that the most impactful "lost" nuances are restored. Furthermore, the authors noted a synergy with Quantization-Aware Training (QAT); using DWR to initialize a model before QAT results in higher final accuracy than standard training paths.
While currently focused on weight-only quantization, future work aims to extend DWR to activation quantization and more complex architectures like Mixture-of-Experts (MoE). DWR stands as a highly efficient, plug-and-play solution for maximizing the intelligence of compressed LLMs without sacrificing deployment speed.
Summary updated