When building state-of-the-art foundation models, the prevailing recipe has long been simple: make the network deeper. Intuitively, stacking more transformer blocks should scale a model's capacity linearly, allowing it to learn increasingly complex representations.
However, recent structural analyses reveal a surprising counter-intuition: a massive portion of the computation in deep models is entirely redundant. Researchers have repeatedly demonstrated that you can strip away substantial fractions of a model's layers—sometimes up to half or more—without causing the network to collapse.
To understand why this is possible, we have to look at how information flows through deep networks, how the mathematical design of the transformer inadvertently fosters inertia, and how we can exploit this "slow manifold" to build faster, leaner models.
The Root Cause: Residual Streams and Identity Transformations
Why do deep models tolerate the sudden removal of their internal organs? The answer lies in the fundamental backbone of the transformer architecture: the residual connection.
In a standard transformer block, the output of layer $l$ is defined as: $$x_{l+1} = x_l + \text{Block}(x_l)$$ where $\text{Block}(x_l)$ represents the combined operations of self-attention and feed-forward networks (FFNs). Because the input $x_l$ is directly added to the output, the network does not need to re-learn the entire representation at every stage. Instead, each layer only needs to compute an incremental update to the ongoing residual stream.
As models grow deeper, these updates often become vanishingly small. As noted by Song et al., this architecture naturally causes successive blocks to produce highly similar outputs, meaning individual layers contribute only incremental changes rather than fundamental transformations. When a layer's mathematical output $\text{Block}(x_l) \approx 0$, the layer effectively defaults to an identity transformation—it passes the input along unchanged.
This phenomenon is not just an accidental byproduct of over-parameterization; it is also driven by standard training dynamics. Li et al. point out that common optimization choices, such as Pre-Layer Normalization (Pre-LN), can inadvertently create "lazy deep layers" that fail to utilize the model's full capacity, leaving the deepest layers structurally under-utilized. Ma et al. formalize this behavior in kernel space, showing that models frequently enter a state of computational redundancy where representations evolve along a highly predictable, slow-moving manifold.
Quantifying the Inertia
To exploit this passivity, we must first measure it. How do we distinguish a critical computational hub from a layer that is merely coasting on the residual stream?
Several independent lines of research have converged on metrics to quantify this layer-wise inertia:
- Block Influence (BI) and Layer Redundancy (LR) Scores: Men et al. introduced Block Influence to evaluate how much a given layer alters the data representation during a forward pass. Similarly, Dumitru et al. developed a Layer Redundancy score to flag blocks operating near identity mappings. Both approaches reveal that a significant percentage of layers perform near-identity transformations.
- Low-Dimensional Trajectories: By treating the internal states of Large Language Models (LLMs) as a dynamical system, Sarfati et al. demonstrated that token embeddings travel along a tightly constrained, low-dimensional manifold. In fact, retaining only the top 25% of principal components at each layer is sufficient to recover the final output probability distribution, proving that the vast majority of the latent space is structurally redundant.
[Input Token] ---> [Cornerstone Layer] ---> [Redundant Layer] ---> [Redundant Layer] ---> [Output]
(Heavy (Near-Identity (Near-Identity
Transformation) Transformation) Transformation)
Crucially, this structural redundancy is a universal property of deep foundation models, transcending modality. It has been documented extensively in Vision Transformers by Balasubramanian et al., in Large Vision-Language Models during their intermediate "stabilization phases" by Wang et al., and even in Time Series Foundation Models by Wilinski et al..
Not All Layers Are Equal
If so many layers are redundant, why not delete them all? The catch is that redundancy is highly non-uniform across the depth of the network.
Zhang et al. discovered that models rely on a small subset of "cornerstone layers"—typically situated early in the architecture—that are absolutely vital. Removing even a single cornerstone layer causes the model's performance to collapse into random guessing.
Conversely, the deeper layers of a network are often remarkably expendable. Gromov et al. demonstrated that up to half of the deepest layers in prominent models like Llama-2 can be deleted with negligible impact on factual knowledge retrieval. This suggests that core semantic concepts and factual knowledge are primarily consolidated in the shallower-to-middle layers, while the deepest layers perform minor, highly redundant refinements.
This effect becomes even more pronounced when general-purpose models are adapted to narrow, specialized tasks. For instance, Ibanez-Lissen et al. showed that for structural code analysis tasks, up to 72% of an LLM's layers can be safely pruned, indicating that deep, generalized linguistic layers contribute almost no task-specific value.
Exploiting the Slow Manifold
Recognizing that deep layers operate on a slow manifold allows us to rethink model compression and inference acceleration. Instead of optimizing individual weight matrices, we can prune or bypass entire structural blocks.
Static Block Pruning
Techniques like SLS (Han et al.) and SLEB (Song et al.) identify and permanently drop redundant blocks post-training. Because redundancy tends to cluster spatially in consecutive sequences rather than being scattered randomly, Chen et al. proposed replacing these excised blocks with a single, highly compressed "lightweight network" to smoothly bridge the gap in the residual stream, maintaining architectural integrity without the cost of the original layers.
Dynamic Bypassing
We do not always have to permanently delete layers. In speculative decoding, a smaller "draft" model predicts tokens quickly, and a larger model verifies them. Xia et al. introduced SWIFT, an on-the-fly speculative decoding framework that requires no auxiliary models. Instead, it dynamically skips intermediate layers during inference to create a temporary, shallow draft model out of the full network, accelerating generation while perfectly preserving output quality. Similarly, during fine-tuning, frameworks like DEFT (Runwal et al.) encourage block-level skipping to minimize activation density and save runtime memory.
Component-Specific Redundancy
Redundancy also exists within the sub-components of individual layers. Jaiswal et al. isolated the feed-forward networks (FFNs) of transformers, showing that the FFNs in middle layers exhibit extreme structural redundancy. By decoupling these FFNs from the self-attention residual stream, they demonstrated that these parameters can be aggressively compressed or offloaded to VRAM lookups with minimal loss in core capabilities.
Ultimately, the pervasive presence of structural redundancy reveals that our current training paradigms are inefficient at utilizing network depth. Until architectures can natively enforce high-information density across all layers, identifying and bypassing the "slow manifolds" of deep networks remains one of our most potent levers for practical model compression.
Key Foundational Work
Quantization-based language model compression (2001) — contributed 2 concept elements to this direction.
Truncation on Combined Word-Based and Class-Based Language Model Using Kullback-Leibler Distance Criterion (1997) — contributed 1 concept elements to this direction.