When we think about deep neural networks, we often imagine a sequential pipeline where each layer builds strictly and critically upon the output of the previous one. In this mental model, removing a single layer should cause the entire pipeline to collapse. However, empirical research into Transformer architectures has revealed a completely different reality: Transformers exhibit highly non-uniform sensitivity across their depth.
While certain "cornerstone" layers are absolutely vital to a model's survival, large swaths of intermediate layers are remarkably redundant. Understanding this non-uniform sensitivity profile allows us to aggressively prune, skip, or compress Transformer layers without destroying their underlying intelligence.
The U-Shaped Importance Curve
If you measure how much a Transformer relies on any given layer, a clear pattern emerges: a characteristic U-shaped importance curve. The initial layers and the final layers of the network are highly sensitive to modification, while the middle-to-deep sections tolerate aggressive parameter removal.
As demonstrated by Jeremias Bohn et al. in Adaptive Parameter Compression for Language Models, the first and last layers are the most critical for task performance, meaning compression ratios must be distributed unevenly to preserve accuracy. This phenomenon is not unique to text. In the domain of generative vision, Enze Xie and Junsong Chen show in SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer that diffusion transformers exhibit the exact same U-shaped curve. The initial layers handle distribution shifts, the final layers sharpen fine details, and the middle blocks remain largely redundant.
Why does this happen? The intuition lies in what these stages are doing:
- The Bottom (Initial Layers): These layers perform low-level feature extraction, token logic, and early structural alignment. If you corrupt these layers, the error propagates and compounds through the rest of the network.
- The Top (Final Layers): These layers map abstract internal representations back to task-specific outputs (like next-token prediction or classification heads). Altering them directly degrades the model's final reasoning output.
- The Middle (Intermediate Layers): These layers act as an informational highway, performing incremental refinements. Because their representations change slowly, they are highly resilient to pruning.
This insight allows for dynamic execution strategies. For instance, Weizhi Wang et al. introduce DiffSkip: Differential Layer Skipping in Large Language Models, which leverages this U-shaped utilization pattern to dynamically skip intermediate blocks on a per-token basis during inference.
Cornerstone Layers and the Residual Stream
To understand why middle layers can be removed so easily, we have to look at the Transformer's backbone: the residual stream. In a standard Transformer block, the output of a layer is not just passed to the next; it is added back to the incoming input:
$$\mathbf{x}_{l} = \mathbf{x}_{l-1} + \text{SubLayer}(\mathbf{x}_{l-1})$$
Because of this additive architecture, each sub-layer (whether it is self-attention or a feed-forward network) computes a residual update ($\Delta \mathbf{x}$) rather than a completely new representation. If a layer is highly redundant, its update is small or highly aligned with the existing residual stream. Removing it behaves like an emulated identity operation.
However, not all layers in the middle are completely useless. Yang Zhang et al. discover in Investigating Layer Importance in Large Language Models that models contain specific "cornerstone layers." While you can drop most intermediate layers with negligible impact, removing just one cornerstone layer causes the model to collapse into random guessing.
Traditional depth pruning often treats entire transformer blocks as indivisible units. As argued by Shoaib Ahmed Siddiqui et al. in A deeper look at depth pruning of LLMs, this coarse-grained approach accidentally removes essential logic components alongside redundant ones. By analyzing sensitivity at the individual sub-layer level, we can strip away redundant blocks while leaving cornerstone layers intact as structural anchors.
Sub-Layer Heterogeneity: Attention vs. Feed-Forward
The sensitivity of a Transformer is not just a function of its depth; it also depends heavily on the type of sub-layer being targeted. A Transformer block consists of two primary engines: Multi-Head Attention (MHA) and the Feed-Forward Network (FFN). They exhibit entirely different redundancy profiles.
Attention Redundancy in Deep Layers
Multi-head attention mechanisms are designed to capture global context and token relationships. However, as a model grows deeper, many of these heads become highly specialized or entirely redundant.
In their seminal work, Elena Voita et al. demonstrate in Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned that a small minority of specialized heads do the bulk of the work, allowing up to 75% of encoder heads to be removed safely. Furthermore, Chonghan Lee et al. show in Token and Head Adaptive Transformers for Efficient Natural Language Processing that this redundancy is concentrated in the later stages of the network, where nearly half of the attention heads can be stripped away without dropping task accuracy.
Feed-Forward Networks as Resilient Targets
While attention heads model token-to-token interactions, FFNs act as key-value memories that process token representations individually. Because FFNs scale up the internal hidden dimension significantly, they contain a massive share of the model's total parameters—and a massive amount of redundancy.
Jeffery Kinnison et al. establish in Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation that FFN sub-layers are far more resilient to neuron removal than attention mechanisms. This is supported by Li Lyna Zhang et al. in SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance, which demonstrates that aggressively pruning FFN dimensions while maintaining attention structures yields massive computational speedups with minimal performance degradation.
Why Local Pruning Preserves Global Stability
A major breakthrough in modern network compression is the ability to prune massive models in a single "one-shot" pass without expensive global retraining. The reason this works comes down to a fundamental property of Transformer layers: local input-output stability.
In SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot, Elias Frantar and Dan Alistarh prove that minimizing the local reconstruction error of each individual layer is mathematically sufficient to maintain global model accuracy. You do not need to optimize the entire network end-to-end to remove weights safely.
Because the residual stream bounds how much an error can propagate, keeping the input-output relationship of a layer stable ensures that subsequent layers receive the representations they expect. By identifying the layer-wise sensitivity distribution beforehand, compression algorithms can preserve sensitive anchor layers, isolate local changes, and aggressively clean out the redundant intermediate blocks that clog up hardware memory and compute.
Key Foundational Work
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior (2020) — contributed 1 concept elements to this direction.
Reducing Transformer Depth on Demand with Structured Dropout (2020) — contributed 1 concept elements to this direction.
Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (2019) — contributed 1 concept elements to this direction.