Identifying Redundant Layers through Activation Similarity

46 concepts40 papers1 proposal

Modern deep neural networks, particularly Large Language Models (LLMs) and Vision Transformers (ViTs), achieve state-of-the-art performance by stacking dozens or even hundreds of layers. However, this extreme depth raises a fundamental question: does every single layer perform unique, necessary computation?

A growing body of research demonstrates that many layers are surprisingly redundant, acting essentially as expensive identity functions. By measuring the similarity between the inputs and outputs of individual network blocks, researchers can identify and eliminate these redundant layers without requiring expensive retraining or fine-tuning.

The Intuition of the Near-Identity Layer

To understand why similarity-based layer pruning works, we must look at the internal mechanics of modern architectures like the Transformer. These networks rely heavily on residual connections, where the output of a block is defined as:

$$\mathbf{h}_{out} = \mathbf{h}_{in} + \text{SubLayer}(\mathbf{h}_{in})$$

If a sub-layer (such as an attention block or a feed-forward network) contributes very little unique information, its output $\text{SubLayer}(\mathbf{h}_{in})$ will be negligible or highly aligned with the input. In this scenario, $\mathbf{h}_{out}$ becomes nearly identical to $\mathbf{h}_{in}$. The model is spending billions of floating-point operations (FLOPs) to pass a hidden state through a layer that barely changes it.

As shown by Men et al. in ShortGPT: Layers in Large Language Models are More Redundant Than You Expect and He et al. in What Matters in Transformers? Not All Attention is Needed, we can formalize this observation by calculating the similarity between the input and output tensors of a block. If the similarity is exceptionally high, the block performs a near-identity transformation and can be targeted for removal with minimal impact on the model's overall representational capacity.

Why Cosine Similarity Captures Semantic Redundancy

When quantifying how much a layer transforms data, a naive approach might measure the Euclidean distance between $\mathbf{h}_{in}$ and $\mathbf{h}_{out}$. However, Euclidean distance is highly sensitive to absolute vector magnitudes, which can fluctuate wildly due to layer normalization or scaling factors without altering the underlying meaning.

Instead, the field has converged on cosine similarity and angular distance as the primary metrics for functional redundancy. As Shopkhoev et al. explain in ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization, cosine distance is uniquely effective because it focuses entirely on the directional alignment of activations rather than their magnitude. In high-dimensional representation spaces, semantic concepts are encoded as directions; if a layer does not rotate or shift the representation vector into a new directional path, it has not altered its semantic content.

By passing a small calibration dataset through the network, researchers can compute these directional shifts. Wang et al. demonstrate in CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information that this angular distance serves as an incredibly accurate proxy for a block's functional contribution, allowing practitioners to pinpoint exactly which layers are safe to drop.

Static Pruning vs. Dynamic Skipping

Once redundant layers are identified via activation similarity, this insight can be applied in two distinct ways: static structural pruning or input-adaptive dynamic skipping.

Static Structural Pruning

In static pruning, the goal is to permanently compress the model before deployment. Chen et al. in Streamlining Redundant Layers to Compress Large Language Models and Yang et al. in EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models use offline calibration to calculate the average input-output similarity across entire layers. Layers that consistently exhibit high similarity are permanently excised from the network stack. Because the pruning rates are allocated non-uniformly based on actual layer importance—as explored by Guo et al. in SlimLLM: Accurate Structured Pruning for Large Language Models—the model maintains its core reasoning capabilities while shrinking significantly in size.

Dynamic Skipping

Not all inputs require the same amount of computational depth; a simple token or image patch may only need a few layers of processing, while a complex one requires the full network. Tan et al. in DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs and Jaiswal et al. in FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping leverage cosine similarity as a real-time, runtime signal. If an early routing mechanism or inline check determines that a block's input and output representations are highly similar for the current token, the model dynamically bypasses that block on the fly, saving computational resources during inference without altering the static architecture.

A Universal Proxy for Model Sensitivity

The beauty of similarity-based importance scoring is its versatility. Because it measures the preservation of representational intent, it serves as a foundational tool across many different model compression paradigms beyond simple layer deletion:

By shifting the focus from weight magnitudes to the geometric alignment of hidden states, similarity-based analysis provides a highly reliable, data-driven window into the inner workings of deep networks, proving that often, less is truly more.

Key Foundational Work

Neuron Merging: Compensating for Pruned Neurons (2020) — contributed 1 concept elements to this direction.
Mitigating Face Recognition Bias via Group Adaptive Classifier (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_similarity_based_layer_pruning_7a7657f1,
  title = {similarity-based layer pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-layer-pruning/rdr_7a7657f1d5354129bed125b2dd97d7d1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_similarity_based_layer_pruning_7a7657f1_v1,
  title = {similarity-based layer pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-layer-pruning/rdr_7a7657f1d5354129bed125b2dd97d7d1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake