TL;DR
DictLLM is a novel post-training compression framework that decomposes large language model (LLM) weights into shared "dictionary" kernels and layer-specific sparse coefficients. By combining parameter sharing, Hessian-aware pruning, and 4-bit quantization, it achieves a massive 5.1× compression of LLaMA-7B with minimal performance loss, outperforming existing low-rank methods by over 3× in compression efficiency.
What Problem Does This Paper Address?
Deploying LLMs on mobile devices is a nightmare for engineers. The primary bottleneck isn't raw compute power; it's memory. Billions of parameters simply won't fit on a phone, and even if they did, moving those weights from memory to the GPU consumes far more energy than the actual math.
Existing solutions like low-rank approximation (shrinking matrices), pruning (removing connections), and quantization (using fewer bits) are usually applied in isolation. While "parameter sharing" (reusing weights across layers) is powerful, it typically requires training a model from scratch—a non-starter for billion-parameter models that have already been pre-trained at great expense. DictLLM bridges this gap by providing a way to apply these techniques together in a "post-training" setting, meaning you can compress a pre-existing model without needing a supercomputer for retraining.
Methods
The authors propose a "3+1" optimization pipeline that transforms standard weight matrices into a more compact, shared representation.
- Activation-Aware Factorization: Instead of just looking at the weights, DictLLM looks at how the weights interact with real data (activations). It uses Singular Value Decomposition (SVD) to decompose grouped layer weights into a single shared dictionary ($D^g$) and layer-specific coefficients ($C^\ell$).
- Dictionary-Aware Hessian Pruning: The coefficients are then "pruned" (set to zero) to reach a target sparsity. Crucially, they use a "dictionary-aware Hessian" to decide which values to cut. Because this math happens in the smaller "coefficient space" rather than the original high-dimensional weight space, it is significantly faster and more computationally efficient.
- Dictionary Refitting: Since the initial dictionary was optimized for dense coefficients, the authors "refit" the shared dictionary to better match the new sparse coefficients. This step recovers much of the accuracy lost during pruning without increasing the model size.
- Hessian-Aware Quantization: Finally, both the dictionary and the sparse coefficients are quantized to a 4-bit block floating-point format (NVFP4). They use Hessian information again here to minimize the error introduced by using lower precision.
Datasets
The researchers evaluated DictLLM across a wide range of standard benchmarks:
- Language Modeling: WikiText-2, Penn TreeBank (PTB), and the Colossal Clean Crawled Corpus (C4).
- Zero-Shot Reasoning: Seven benchmarks including OpenBookQA, WinoGrande, HellaSwag, PIQA, MathQA, ARC-Easy, and ARC-Challenge.
- Models: LLaMA-7B, LLaMA2-7B, OPT-6.7B, Mistral-7B, and the Qwen3 family (1.7B to 14B).
Key Insights and Ideas
- The Power of Sharing: The authors found that dictionaries store "shared information" reused across layers, while coefficients handle "layer-specific adaptation." This justifies keeping the dictionary dense while aggressively pruning the coefficients.
- Efficiency in the Coefficient Space: By projecting the Hessian (a matrix of second-order derivatives used to estimate parameter importance) into the coefficient space, the computational cost of pruning drops from $O(d_{in}^3)$ to $O(k^3)$, where $k$ is the much smaller dictionary size.
- Refitting is the "Secret Sauce": Pruning naturally introduces error. By simply re-solving a least-squares problem to update the shared dictionary after pruning, the model recovers significant performance (e.g., a 1.04 point improvement in perplexity at 50% sparsity).
Key Results
DictLLM consistently pushes the Pareto frontier of model size versus performance:
- Extreme Compression: It compressed LLaMA-7B by 5.1× with only a 2.9-point increase in WikiText-2 perplexity. For context, the previous state-of-the-art low-rank method achieved a much lower compression ratio for the same performance level.
- Superior Accuracy: At 2× compression, DictLLM achieved an average zero-shot reasoning accuracy of 0.43, significantly higher than the 0.36 achieved by the "Basis Sharing" baseline.
- Real-World Speedup: On an NVIDIA RTX 6000 Blackwell GPU, DictLLM reached a decoding speed of 126.81 tokens/s, a 1.24× increase over the original uncompressed model, while reducing peak memory usage by 1.7×.
Limitations and Future Work
The main limitation acknowledged is that coefficient pruning produces unstructured sparsity. While this is great for reducing the number of parameters stored, it requires specialized software kernels (like the MACKO library used in the paper) to translate that sparsity into actual hardware speedups on standard GPUs.
The authors suggest that exploring structured coefficient sparsity (like block sparsity) could be a promising future direction to make the compressed models even more "hardware-friendly" and easier to deploy on a wider variety of mobile chips.
Summary updated