Efficient Compression and Structural Pruning for Large Language Models

This direction is split into other directions: edge deployment constraints (resource-constrained model deployment), model deployment constraints. Its last version stays readable and citable here.

Version 1 of 1372 papers384 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are defined by their scale, but that same scale makes them nearly impossible to deploy on consumer-grade hardware. A model with 70 billion parameters requires roughly 140GB of VRAM just to load in half-precision—far exceeding the capacity of a standard GPU.

To bridge this gap, researchers are developing techniques to shrink these models without losing the "parametric knowledge" they've acquired during pre-training. This research direction focuses on three main pillars: ultra-low-bit quantization, structural pruning that respects hardware constraints, and "activation-aware" methods that prioritize the most influential parts of the network.

Moving Beyond Simple Rounding with Ultra-Low Quantization

The most direct way to compress a model is quantization—reducing the precision of its weights. While moving from 16-bit to 8-bit or 4-bit is now standard, the current frontier is "sub-4-bit" and even "sub-1-bit" compression.

Simple rounding at these extremes causes catastrophic accuracy loss. To solve this, researchers like Vage Egiazarian and Andrei Panferov introduced AQLM (Additive Quantization of Language Models). Instead of rounding individual numbers, AQLM uses multi-codebook quantization—essentially finding a small set of "representative vectors" that can be added together to reconstruct the original weight matrix.

Other methods, such as DB-LLM by Hong Chen and Haotong Qin, use Dual-Binarization (Accurate Dual-Binarization for Efficient LLMs). This approach decomposes 2-bit weights into two independent 1-bit sets. By treating quantization as a decomposition problem rather than a precision problem, these methods allow 2-bit models to occasionally outperform full-precision models that are significantly smaller in total size.

Structural Pruning and the Dependency Problem

Pruning involves removing redundant parameters entirely. However, "unstructured" pruning (removing random individual weights) creates sparse matrices that are actually slower on modern GPUs, which prefer dense, contiguous blocks of data.

The challenge with "structural" pruning (removing entire rows, columns, or attention heads) is that LLMs have complex internal dependencies. If you prune a dimension in one layer, it must match the input dimension of the next. Xinyin Ma and Gongfan Fang addressed this in LLM-Pruner (On the Structural Pruning of Large Language Models) by using dependency graphs to ensure the model remains "executable" after parts are removed.

To recover the knowledge lost during pruning without retraining on trillions of tokens, frameworks like LoRAShear (Efficient Large Language Model Structured Pruning and Knowledge Recovery) by Tianyi Chen and Tianyu Ding use a "knowledge-transfer" algorithm. They identify which parameters are most salient and use Low-Rank Adaptation (LoRA) to quickly "heal" the pruned model on a tiny fraction of the original data.

Activation-Awareness: Prioritizing the Outliers

A key insight in modern compression is that not all weights are created equal. In Transformers, a small number of "outlier" activations often carry the most significant information. If a weight interacts with a massive activation value, even a tiny error in that weight will be amplified.

ASVD (Activation-aware Singular Value Decomposition), developed by Zhihang Yuan and colleagues, scales weight matrices based on their typical activation magnitudes before performing SVD. This ensures that the decomposition preserves the "heavy hitters" in the feature space. Similarly, OATS (Outlier-Aware Pruning Through Sparse and Low Rank Decomposition) by Vardan Papyan decomposes weights into a sum of a sparse matrix (to capture outliers) and a low-rank matrix (to capture the general structure), achieving high compression with minimal accuracy drops.

Hybrid Tuning and Distillation

The most recent trend is moving away from "compress then fine-tune" toward simultaneous optimization. Pruning-Aware Tuning (PAT) (Pruning-Aware Tuning for Large Language Models) by Huanrui Yang integrates sparsification directly into the fine-tuning process. This allows the model to adapt its remaining weights to the sparse structure in real-time.

Furthermore, Knowledge Distillation (KD) is being reimagined for the LLM era. Instead of just matching the final output of a large "teacher" model, methods like CoDIR (Contrastive Distillation on Intermediate Representations) by Siqi Sun and Zhe Gan use contrastive learning to teach a small student model the complex structural patterns hidden in the teacher's internal layers. This ensures that the compressed model doesn't just parrot the right answers but actually mimics the teacher's internal reasoning logic.

Papers in this version

Showing 60 of 372 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_compression_and_structural_pru_c67556db,
  title = {Efficient Compression and Structural Pruning for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-compression-and-structural-pruning-for-large-language-models/rdr_c67556db37c942ebb6fd91e24a5e0bb2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_compression_and_structural_pru_c67556db_v1,
  title = {Efficient Compression and Structural Pruning for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-compression-and-structural-pruning-for-large-language-models/rdr_c67556db37c942ebb6fd91e24a5e0bb2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}