Standard Transformer architectures operate on a rigid principle of uniform computation: every token in a sequence, from a simple comma to a highly complex technical term, passes through the exact same sequence of layers, attention heads, and feed-forward networks. This token-agnostic processing creates a massive computational bottleneck. In practice, textual and visual data possess highly variable information density. Treating every input with identical intensity wastes significant processing power on simple or redundant tokens while capping the network's ability to allocate deeper reasoning to difficult ones.
To resolve this inefficiency, a prominent research direction focuses on adaptive computational allocation. By decoupling a model's computational depth and width from the raw sequence length, these techniques dynamically route resources to tokens based on their semantic complexity.
The Inefficiency of Fixed-Depth Processing
In a conventional Transformer, the computational cost scales linearly with the number of layers and quadratically with sequence length, regardless of the task's actual difficulty. This structure assumes that every token requires the same amount of contextualization at every step of the network.
However, linguistic and visual features do not update at a uniform frequency. For instance, high-frequency function words or background image patches reach semantic saturation early in the network. Forcing these "easy" tokens to undergo the full depth of a model dilutes the hidden states and wastes FLOPs.
To visualize why adaptive allocation works, consider how information propagates through a network. Early layers typically extract low-level syntactic or structural features, while deeper layers resolve complex semantic dependencies and abstract reasoning. Adaptive architectures exploit this by allowing tokens to exit the computational graph as soon as their representations stabilize, or by skipping layers that add marginal utility.
Dynamic Depth via Early Exiting and Layer Skipping
One of the most direct ways to break uniform processing is to vary the model's depth dynamically per token or per sequence. Instead of executing the entire stack of layers, the model evaluates whether a token's representation is already sufficient to make an accurate prediction.
Standard Transformer: [Token] -> Layer 1 -> Layer 2 -> Layer 3 -> Layer 4 -> Output
Early-Exit (CALM): [Token] -> Layer 1 -> Layer 2 -> (Confident?) -> Output
Layer Skipping: [Token] -> Layer 1 -> [Skip Layer 2/3] -> Layer 4 -> Output
Token-Level Early Exiting
In frameworks like CALM, introduced by Tal Schuster et al. in Confident Adaptive Language Modeling, the model dynamically decides how many layers to execute for each generated token. By measuring the confidence of intermediate predictions (using metrics like logit entropy), the model can exit early on straightforward tokens, reserving the full stack for ambiguous or context-heavy terms. Similarly, AdaInfer, proposed by Siqi Fan et al. in Not All Layers of LLMs Are Necessary During Inference, applies lightweight statistical classifiers to monitor logit confidence at intermediate layers, terminating inference early on simpler queries without modifying the underlying model weights.
Layer Skipping and Recurrent Blocks
Rather than exiting completely, some architectures allow tokens to selectively bypass specific intermediate layers. Because Transformers rely heavily on residual connections, a token can skip a layer's computation simply by routing its current representation directly around the attention or feed-forward blocks. The authors of What Layers When: Learning to Skip Compute in LLMs with Residual Gates demonstrate this by adding differentiable gates to the residual stream, allowing the model to learn exactly which layers are redundant for specific tokens.
Alternatively, models can share parameters across a single block and loop tokens through it recursively. Sangmin Bae et al. develop this concept in Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation, using a lightweight router to determine the exact number of recursive passes each token requires through a shared block. This allows the model to scale its test-time reasoning dynamically based on token difficulty.
Token Dropping and Sequence Skimming
While early exiting modifies the vertical depth an individual token traverses, token dropping and skimming modify the horizontal sequence length as it moves through the network. In many tasks, such as long-document analysis or multi-turn dialogue, the relevance of specific tokens diminishes as higher-level context is established.
Layer 1: [The] [quantum] [computing] [system] [is] [fast] (6 tokens)
\ | | / / /
Layer 2: --- [quantum] [computing] [system] --- --- (3 tokens)
Pruning Redundant Tokens
As representations pass through intermediate layers, they tend to converge and become highly correlated. Yue Guan et al. exploit this in Transkimmer: Transformer Learns to Layer-wise Skim by introducing a differentiable reparameterization mechanism that identifies and drops redundant tokens layer-by-layer. By removing low-information tokens early, subsequent layers only compute attention over a compressed, highly informative subset of the original sequence.
This principle is equally effective during model training. For example, Le Hou et al. demonstrate in Token Dropping for Efficient BERT Pretraining that dropping "easy" tokens (identified by low masked language modeling loss) in intermediate layers allows the model to focus its gradient updates and computational budget on contextually dense terms, accelerating pretraining without sacrificing downstream accuracy.
Contextual Elimination in Dialogue
In multi-turn conversations, historical context often contains substantial noise. Jianxin Liang et al. address this in Attend, Select and Eliminate: Accelerating Multi-turn Response Selection with Dual-attention-based Content Elimination. Their architecture progressively eliminates irrelevant words from the dialogue history across layers, ensuring that the heavy cross-attention mechanisms in deeper layers only process tokens directly relevant to selecting the correct response.
Dynamic Width and Modular Routing
Beyond adjusting depth or sequence length, adaptive allocation can alter the internal dimensions or specific modules a token activates. This can be thought of as adjusting the "width" or the specialized routing of the network on a per-token basis.
Modular Assemblies
Instead of forcing every token to follow a strict, sequential layer order from $1$ to $N$, architectures can treat the Transformer as a flexible pool of modules. Zhuocheng Gong et al. propose this paradigm shift in Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules. Their Mixture-of-Modules (MoM) approach allows tokens to dynamically "jump" between different attention and feed-forward modules based on their immediate processing needs, breaking the rigid layer-by-layer convention entirely.
Granular Width and Dimension Sparsity
Standard Transformers allocate identical hidden state dimensions to all tokens. However, not all dimensions contribute equally to every token's representation. Yilong Chen addresses this in Mixture of Hidden-Dimensions: Not All Hidden-States’ Dimensions are Needed in Transformer by splitting the hidden space into a set of shared dimensions for common features and specialized dimensions that are dynamically routed per token.
Similarly, Alessio Devoto et al. introduce Adaptive Computation Modules (ACMs) in Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference, which replace standard wide layers with a sequence of smaller sub-learners and a gating mechanism. The model dynamically activates only the necessary number of sub-learners for a given token, effectively adjusting the layer's width on the fly.
Why It Works: The Intuition of Variable Information Density
The success of adaptive computational allocation stems from a fundamental property of natural data: information is non-uniformly distributed.
In text, words like "the," "is," or punctuation marks serve structural purposes but carry minimal semantic surprise. Once their local syntactic role is resolved in the first few layers, passing them through dozens more layers yields diminishing marginal utility. Conversely, rare nouns, ambiguous verbs, or complex logical steps require extensive contextualization across the entire sequence to be accurately resolved.
By breaking the rigid token-by-layer grid, adaptive computation aligns the model's computational graph with the underlying data's information density. Simple tokens are processed quickly via early exits, layer skipping, or narrow hidden dimensions, freeing up the model's capacity and memory bandwidth. This allows the architecture to focus its heavy-duty parameters and deep attention mechanisms precisely where semantic uncertainty is highest.
Key Foundational Work
Character-level Language Modeling with Gated Hierarchical Recurrent Neural Networks (2018) — contributed 1 concept elements to this direction.
Chunk-based Decoder for Neural Machine Translation (2017) — contributed 1 concept elements to this direction.