Modern deep neural networks, particularly Large Language Models (LLMs) and Vision Transformers (ViTs), achieve state-of-the-art performance by stacking dozens or even hundreds of layers. However, this extreme depth raises a fundamental question: does every single layer perform unique, necessary computation?
A growing body of research demonstrates that many layers are surprisingly redundant, acting essentially as expensive identity functions. By measuring the similarity between the inputs and outputs of individual network blocks, researchers can identify and eliminate these redundant layers without requiring expensive retraining or fine-tuning.
The Intuition of the Near-Identity Layer
To understand why similarity-based layer pruning works, we must look at the internal mechanics of modern architectures like the Transformer. These networks rely heavily on residual connections, where the output of a block is defined as:
$$\mathbf{h}_{out} = \mathbf{h}_{in} + \text{SubLayer}(\mathbf{h}_{in})$$
If a sub-layer (such as an attention block or a feed-forward network) contributes very little unique information, its output $\text{SubLayer}(\mathbf{h}_{in})$ will be negligible or highly aligned with the input. In this scenario, $\mathbf{h}_{out}$ becomes nearly identical to $\mathbf{h}_{in}$. The model is spending billions of floating-point operations (FLOPs) to pass a hidden state through a layer that barely changes it.
As shown by Men et al. in ShortGPT: Layers in Large Language Models are More Redundant Than You Expect and He et al. in What Matters in Transformers? Not All Attention is Needed, we can formalize this observation by calculating the similarity between the input and output tensors of a block. If the similarity is exceptionally high, the block performs a near-identity transformation and can be targeted for removal with minimal impact on the model's overall representational capacity.
Why Cosine Similarity Captures Semantic Redundancy
When quantifying how much a layer transforms data, a naive approach might measure the Euclidean distance between $\mathbf{h}_{in}$ and $\mathbf{h}_{out}$. However, Euclidean distance is highly sensitive to absolute vector magnitudes, which can fluctuate wildly due to layer normalization or scaling factors without altering the underlying meaning.
Instead, the field has converged on cosine similarity and angular distance as the primary metrics for functional redundancy. As Shopkhoev et al. explain in ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization, cosine distance is uniquely effective because it focuses entirely on the directional alignment of activations rather than their magnitude. In high-dimensional representation spaces, semantic concepts are encoded as directions; if a layer does not rotate or shift the representation vector into a new directional path, it has not altered its semantic content.
By passing a small calibration dataset through the network, researchers can compute these directional shifts. Wang et al. demonstrate in CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information that this angular distance serves as an incredibly accurate proxy for a block's functional contribution, allowing practitioners to pinpoint exactly which layers are safe to drop.
Static Pruning vs. Dynamic Skipping
Once redundant layers are identified via activation similarity, this insight can be applied in two distinct ways: static structural pruning or input-adaptive dynamic skipping.
Static Structural Pruning
In static pruning, the goal is to permanently compress the model before deployment. Chen et al. in Streamlining Redundant Layers to Compress Large Language Models and Yang et al. in EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models use offline calibration to calculate the average input-output similarity across entire layers. Layers that consistently exhibit high similarity are permanently excised from the network stack. Because the pruning rates are allocated non-uniformly based on actual layer importance—as explored by Guo et al. in SlimLLM: Accurate Structured Pruning for Large Language Models—the model maintains its core reasoning capabilities while shrinking significantly in size.
Dynamic Skipping
Not all inputs require the same amount of computational depth; a simple token or image patch may only need a few layers of processing, while a complex one requires the full network. Tan et al. in DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs and Jaiswal et al. in FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping leverage cosine similarity as a real-time, runtime signal. If an early routing mechanism or inline check determines that a block's input and output representations are highly similar for the current token, the model dynamically bypasses that block on the fly, saving computational resources during inference without altering the static architecture.
A Universal Proxy for Model Sensitivity
The beauty of similarity-based importance scoring is its versatility. Because it measures the preservation of representational intent, it serves as a foundational tool across many different model compression paradigms beyond simple layer deletion:
- Quantization: The authors of Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark use input-output hidden state similarity to predict block sensitivity. Blocks with high similarity can be quantized to aggressive, low bit-widths because their minor errors will not compound drastically across the network.
- Layer Merging: Instead of deleting blocks, the authors of LaCo: Large Language Model Pruning via Layer Collapse use cosine similarity to guide the merging of consecutive layers, ensuring the collapsed representations remain close to the parent model's original trajectory.
- Parameter-Efficient Fine-Tuning: The authors of Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models track the similarity between modified layer activations and their original pre-trained counterparts to identify and freeze redundant adapters early in the fine-tuning process, drastically reducing memory overhead.
By shifting the focus from weight magnitudes to the geometric alignment of hidden states, similarity-based analysis provides a highly reliable, data-driven window into the inner workings of deep networks, proving that often, less is truly more.
Key Foundational Work
Neuron Merging: Compensating for Pruned Neurons (2020) — contributed 1 concept elements to this direction.
Mitigating Face Recognition Bias via Group Adaptive Classifier (2021) — contributed 1 concept elements to this direction.