DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

Jinho Park, Se Young Chun, Mingoo SeokarXiv 2026 · arXiv

TL;DR

DictLLM is a novel post-training compression framework that decomposes large language model (LLM) weights into shared "dictionary" kernels and layer-specific sparse coefficients. By combining parameter sharing, Hessian-aware pruning, and 4-bit quantization, it achieves a massive 5.1× compression of LLaMA-7B with minimal performance loss, outperforming existing low-rank methods by over 3× in compression efficiency.

What Problem Does This Paper Address?

Deploying LLMs on mobile devices is a nightmare for engineers. The primary bottleneck isn't raw compute power; it's memory. Billions of parameters simply won't fit on a phone, and even if they did, moving those weights from memory to the GPU consumes far more energy than the actual math.

Existing solutions like low-rank approximation (shrinking matrices), pruning (removing connections), and quantization (using fewer bits) are usually applied in isolation. While "parameter sharing" (reusing weights across layers) is powerful, it typically requires training a model from scratch—a non-starter for billion-parameter models that have already been pre-trained at great expense. DictLLM bridges this gap by providing a way to apply these techniques together in a "post-training" setting, meaning you can compress a pre-existing model without needing a supercomputer for retraining.

Methods

The authors propose a "3+1" optimization pipeline that transforms standard weight matrices into a more compact, shared representation.

Figure 1: DictLLM: Dictionary-based compression method. DictLLM resolves the weight memory bottleneck by compressing transformer weights into a shareable dictionary. It compounds four compression steps. ➊The first step factorizes input-scaled weight matrices of grouped layers into a shared dictionary and layer-specific coefficients. ➋We then prune the layer-specific coefficients using a dictionary-aware Hessian. ➌We refit the shared dictionary to the resulting sparse coefficients. ➍We quantize both the dictionary and the coefficients into NVFP4 using input statistics for the dictionary and dictionary-aware Hessians for the coefficients. The resulting compressed representation enables efficient projections by transferring a small set of shared dictionaries.
  1. Activation-Aware Factorization: Instead of just looking at the weights, DictLLM looks at how the weights interact with real data (activations). It uses Singular Value Decomposition (SVD) to decompose grouped layer weights into a single shared dictionary ($D^g$) and layer-specific coefficients ($C^\ell$).
  2. Dictionary-Aware Hessian Pruning: The coefficients are then "pruned" (set to zero) to reach a target sparsity. Crucially, they use a "dictionary-aware Hessian" to decide which values to cut. Because this math happens in the smaller "coefficient space" rather than the original high-dimensional weight space, it is significantly faster and more computationally efficient.
  3. Dictionary Refitting: Since the initial dictionary was optimized for dense coefficients, the authors "refit" the shared dictionary to better match the new sparse coefficients. This step recovers much of the accuracy lost during pruning without increasing the model size.
  4. Hessian-Aware Quantization: Finally, both the dictionary and the sparse coefficients are quantized to a 4-bit block floating-point format (NVFP4). They use Hessian information again here to minimize the error introduced by using lower precision.

Datasets

The researchers evaluated DictLLM across a wide range of standard benchmarks:

  • Language Modeling: WikiText-2, Penn TreeBank (PTB), and the Colossal Clean Crawled Corpus (C4).
  • Zero-Shot Reasoning: Seven benchmarks including OpenBookQA, WinoGrande, HellaSwag, PIQA, MathQA, ARC-Easy, and ARC-Challenge.
  • Models: LLaMA-7B, LLaMA2-7B, OPT-6.7B, Mistral-7B, and the Qwen3 family (1.7B to 14B).

Key Insights and Ideas

  • The Power of Sharing: The authors found that dictionaries store "shared information" reused across layers, while coefficients handle "layer-specific adaptation." This justifies keeping the dictionary dense while aggressively pruning the coefficients.
  • Efficiency in the Coefficient Space: By projecting the Hessian (a matrix of second-order derivatives used to estimate parameter importance) into the coefficient space, the computational cost of pruning drops from $O(d_{in}^3)$ to $O(k^3)$, where $k$ is the much smaller dictionary size.
  • Refitting is the "Secret Sauce": Pruning naturally introduces error. By simply re-solving a least-squares problem to update the shared dictionary after pruning, the model recovers significant performance (e.g., a 1.04 point improvement in perplexity at 50% sparsity).

Key Results

DictLLM consistently pushes the Pareto frontier of model size versus performance:

  • Extreme Compression: It compressed LLaMA-7B by 5.1× with only a 2.9-point increase in WikiText-2 perplexity. For context, the previous state-of-the-art low-rank method achieved a much lower compression ratio for the same performance level.
  • Superior Accuracy: At 2× compression, DictLLM achieved an average zero-shot reasoning accuracy of 0.43, significantly higher than the 0.36 achieved by the "Basis Sharing" baseline.
  • Real-World Speedup: On an NVIDIA RTX 6000 Blackwell GPU, DictLLM reached a decoding speed of 126.81 tokens/s, a 1.24× increase over the original uncompressed model, while reducing peak memory usage by 1.7×.

Limitations and Future Work

The main limitation acknowledged is that coefficient pruning produces unstructured sparsity. While this is great for reducing the number of parameters stored, it requires specialized software kernels (like the MACKO library used in the paper) to translate that sparsity into actual hardware speedups on standard GPUs.

The authors suggest that exploring structured coefficient sparsity (like block sparsity) could be a promising future direction to make the compressed models even more "hardware-friendly" and easier to deploy on a wider variety of mobile chips.

Summary updated

Abstract

Efficient deployment of large language models (LLMs) on mobile devices requires balancing performance with compute and memory constraints. Post-training compression is an effective approach to managing this trade-off, while requiring much less computation than in-training counterparts. We propose DictLLM, a dictionary-based compression scheme in which weight matrices within grouped layers are decomposed into a shared dictionary and layer-specific sparse coefficients, directly obtained from a pre-trained model without further training. Specifically, we cast DictLLM as a four-step post-training optimization. First, we obtain shared dictionaries via singular value decomposition (SVD) of input-scaled weight matrices, amortizing rank across grouped layers; second, we sparsify the layer-specific coefficients to the target compression ratio using a dictionary-aware Hessian; and third, we refine the dictionaries given the resulting sparse coefficients. Finally, we aggressively quantize both components using factor-specific Hessian information: input covariance for the dictionaries and the dictionary-aware Hessian for the coefficients. The resulting representation reduces memory footprint by storing a single shared dictionary per layer group, along with sparse layer-specific coefficients. On language modeling and zero-shot reasoning benchmarks, DictLLM consistently outperforms existing post-training low-rank compression methods and pushes the Pareto frontier between model size and performance. DictLLM compresses the original LLaMA-7B model by 5.1x with only a 2.9-point increase in WikiText-2 perplexity, achieving a 3.6x higher compression ratio than the state-of-the-art low-rank method.
BibTeX
@misc{park2026dictllm,
  author = {Jinho Park and Se Young Chun and Mingoo Seok},
  title = {{DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels}},
  year = {2026},
  eprint = {2611.08082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08082},
}
Figure