Non-Uniform Structural Sensitivity in Transformer Compression

This direction is split into other directions: layer-wise pruning sensitivity (transformer layer sensitivity). Its last version stays readable and citable here.

Version 1 of 1104 papers119 conceptsfrozen 2026-10-04

Standard model compression often treats every layer of a neural network as equally expendable. However, a growing body of research reveals that Transformer architectures exhibit a distinct "U-shaped" sensitivity profile. While the middle layers of a model are often highly redundant and can be aggressively pruned, the initial and final layers act as critical structural anchors for feature extraction and task-specific prediction.

The U-Shaped Importance Curve

The most consistent finding across recent compression literature is that the "ends" of a Transformer are more important than its "middle." In Towards Efficient and Elastic Visual Question Answering with Doubly Slimmable Transformer, Zhou Yu and Jun Yu identify this U-shaped distribution, noting that the first and last layers are vital for initial encoding and final output generation.

This observation has led to "central layer pruning" strategies. For instance, Xinyin Ma in LLM-Pruner demonstrates that preserving these sensitive boundary layers allows for much more aggressive compression in the middle without damaging the model's reasoning capabilities. Similarly, the OpenBA-V2 framework by Dan Qiao achieves a 77% compression ratio by prioritizing the removal of intermediate layers while keeping the input and output stages intact.

The intuition is that the middle layers often perform iterative refinement or "over-thinking" that is redundant for many tasks. In diffusion models, Enze Xie and colleagues in SANA 1.5 find that middle blocks can be bypassed because the early layers handle distribution shifts and the final layers handle detail sharpening.

Component-Specific Redundancy

Beyond depth, researchers have found that the internal components of a Transformer—Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN)—respond differently to pruning.

Moving Beyond Uniform Sparsity

If layers aren't equal, pruning shouldn't be uniform. Modern techniques use game theory and adaptive routing to find the "sweet spot" for each layer.

SV-NUP (Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models) by Han Yu uses Shapley values to quantify exactly how much each layer contributes to the model's performance. This allows for non-uniform pruning that reduces perplexity by nearly 20% compared to uniform methods.

Other researchers focus on merging rather than deleting. Sliding-Window Merging (SWM) (Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs) by Xuan Ding identifies "patches" of consecutive layers that perform nearly identical functions and merges them into single layers. This preserves the functional continuity of the network better than simply deleting blocks.

Why It Works: The Information Bottleneck

The success of non-uniform pruning suggests that Transformers naturally develop an internal hierarchy of information. Early layers focus on broad feature extraction, while later layers specialize in task-specific refinement. Rabeeh Karimi Mahabadi explores this in VIBERT (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning), showing that forcing models to be more selective about information flow—essentially creating intentional bottlenecks—improves generalization.

By recognizing that the "middle" of the Transformer is often a high-capacity buffer for information that has already been processed, researchers can strip away massive amounts of computation without losing the core logic residing at the model's boundaries.

Papers in this version

Showing 60 of 104 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_uniform_structural_sensitivity_in_tr_859a524a,
  title = {Non-Uniform Structural Sensitivity in Transformer Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-uniform-structural-sensitivity-in-transformer-compression/rdr_859a524a70424f13955e5bedd815010e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_uniform_structural_sensitivity_in_tr_859a524a_v1,
  title = {Non-Uniform Structural Sensitivity in Transformer Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-uniform-structural-sensitivity-in-transformer-compression/rdr_859a524a70424f13955e5bedd815010e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}