Standard model compression often treats every layer of a neural network as equally expendable. However, a growing body of research reveals that Transformer architectures exhibit a distinct "U-shaped" sensitivity profile. While the middle layers of a model are often highly redundant and can be aggressively pruned, the initial and final layers act as critical structural anchors for feature extraction and task-specific prediction.
The U-Shaped Importance Curve
The most consistent finding across recent compression literature is that the "ends" of a Transformer are more important than its "middle." In Towards Efficient and Elastic Visual Question Answering with Doubly Slimmable Transformer, Zhou Yu and Jun Yu identify this U-shaped distribution, noting that the first and last layers are vital for initial encoding and final output generation.
This observation has led to "central layer pruning" strategies. For instance, Xinyin Ma in LLM-Pruner demonstrates that preserving these sensitive boundary layers allows for much more aggressive compression in the middle without damaging the model's reasoning capabilities. Similarly, the OpenBA-V2 framework by Dan Qiao achieves a 77% compression ratio by prioritizing the removal of intermediate layers while keeping the input and output stages intact.
The intuition is that the middle layers often perform iterative refinement or "over-thinking" that is redundant for many tasks. In diffusion models, Enze Xie and colleagues in SANA 1.5 find that middle blocks can be bypassed because the early layers handle distribution shifts and the final layers handle detail sharpening.
Component-Specific Redundancy
Beyond depth, researchers have found that the internal components of a Transformer—Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN)—respond differently to pruning.
- FFN vs. Attention: Jeremiah Zhe Liu and Zi Lin introduced SNIP (Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior), which pushes redundant FFN layers toward simple identity mappings. They found that FFNs at the extremities are more redundant than those in the middle, but attention heads are generally more over-parameterized than FFNs.
- Activation Variance: In generative tasks, the final layer often exhibits a spike in activation variance. Zachary Novack and Taylor Berg-Kirkpatrick show in Presto! (Presto! Distilling Steps and Layers for Accelerating Music Generation) that preserving this high-variance terminal layer is critical for audio fidelity, even when dropping many intermediate blocks.
- Distributional Differences: James Liu and Han Guo find in TEAL (Training-Free Activation Sparsity in Large Language Models) that hidden states in MLP blocks follow Laplacian distributions, making them more amenable to magnitude-based pruning than the Gaussian-distributed states in attention blocks.
Moving Beyond Uniform Sparsity
If layers aren't equal, pruning shouldn't be uniform. Modern techniques use game theory and adaptive routing to find the "sweet spot" for each layer.
SV-NUP (Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models) by Han Yu uses Shapley values to quantify exactly how much each layer contributes to the model's performance. This allows for non-uniform pruning that reduces perplexity by nearly 20% compared to uniform methods.
Other researchers focus on merging rather than deleting. Sliding-Window Merging (SWM) (Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs) by Xuan Ding identifies "patches" of consecutive layers that perform nearly identical functions and merges them into single layers. This preserves the functional continuity of the network better than simply deleting blocks.
Why It Works: The Information Bottleneck
The success of non-uniform pruning suggests that Transformers naturally develop an internal hierarchy of information. Early layers focus on broad feature extraction, while later layers specialize in task-specific refinement. Rabeeh Karimi Mahabadi explores this in VIBERT (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning), showing that forcing models to be more selective about information flow—essentially creating intentional bottlenecks—improves generalization.
By recognizing that the "middle" of the Transformer is often a high-capacity buffer for information that has already been processed, researchers can strip away massive amounts of computation without losing the core logic residing at the model's boundaries.