Standard neural networks operate on a "fixed-budget" principle. Whether a Transformer is predicting the next word in a simple greeting or solving a complex multi-step math problem, it spends the exact same number of floating-point operations (FLOPs) per token. This rigidity creates a fundamental bottleneck: complex tasks are often "too deep" for the model's fixed architecture, while long sequences suffer from diluted attention or memory constraints.
A growing body of research suggests that computation should instead be elastic, scaling its depth, memory, and "thinking time" dynamically based on the complexity and length of the input.
Expanding Thinking Time via Scratchpads
The most intuitive way to scale computation is to give a model more "room to think" on the page. Anders Andreassen, Maxwell Nye, and their colleagues introduced the scratchpad mechanism in Show Your Work: Scratchpads for Intermediate Computation with Language Models.
In a standard forward pass, a model must map an input to an output in a fixed number of layers. By training the model to output intermediate steps—essentially a sequence of internal thoughts—the researchers effectively decoupled the computational budget from the model's depth. Because the number of FLOPs scales with the length of the output sequence, the model can allocate more processing steps to difficult algorithmic problems, like long addition or code execution, that it could never solve in a single pass.
Elastic Memory and Dynamic Context
While scratchpads scale computation in "time" (sequence steps), other methods scale the model’s "workspace" (the number of vectors it maintains).
Traditional models often use fixed-size memory bottlenecks or windowed attention, which can lead to the "lost in the middle" phenomenon. To counter this, Guanghui Qin and Benjamin Van Durme developed DODO (Dynamic Contextual Compression) in Nugget 2D: Dynamic Contextual Compression for Scaling Decoder-only Language Models. Instead of compressing a context into a fixed number of slots, DODO uses a differentiable mechanism to select a variable number of "nugget" vectors. This allows the model’s representational capacity to grow naturally with the sequence length, preserving more information for dense, complex segments while aggressively compressing redundant data.
Similarly, Fuzhao Xue and Mostafa Dehghani proposed AdaTape in Adaptive Computation with Elastic Input Sequence. AdaTape uses an "Adaptive Tape Reading" algorithm to dynamically pull additional tokens from a memory bank into the input sequence. By expanding the effective context window based on task difficulty, the model can "buy" more computation by processing a longer sequence of tokens when it encounters a challenging input.
Scaling Depth and Receptive Fields
If a sequence is twice as long, does it need twice as many layers to process? Some researchers argue the answer is yes.
In Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation, Ta-Chung Chi and Peter Ramadge introduced RegularGPT, which employs length-adaptive depth scaling. By increasing the number of layers used during inference for longer inputs, the model gains the additional processing steps required for long-range logical reasoning. This allows the model to extrapolate to sequences much longer than those seen during training.
This principle of scaling the architecture to the sequence length appears in other forms as well:
- Logarithmic Scaling: Ruslan Khalitov and Zhirong Yang proposed ChordMixer in ChordMixer: A Scalable Neural Attention Model for Sequences with Different Length, which scales the number of processing blocks to the logarithm of the input length ($\log N$). This ensures a full receptive field for any sequence size without the waste of padding.
- Kernel Scaling: In MambaSL, Yoo-Min Jung and Leekyung Kim scale the kernel size of input projections relative to the sequence length, ensuring the receptive field is always sufficient to capture the necessary context for gating mechanisms.
Nonparametric Bottlenecks
Finally, the bottleneck itself can be made adaptive. Standard Variational Autoencoders (VAEs) use a fixed-size latent vector, which forces a model to compress a 10-page story into the same space as a 10-word sentence.
James Henderson and Fabio Fehr addressed this in A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck. Their Nonparametric Variational Autoencoder (NVAE) allows the latent space to dynamically adjust its size based on the complexity of the input. By leveraging Bayesian nonparametrics, the model scales its representational capacity linearly with the data length, enabling better generalization to out-of-distribution sequence lengths.
Summary
The shift from fixed to adaptive computation represents a move toward more "biological" processing. By scaling output length (Andreassen et al.), input context (Xue et al.), model depth (Chi et al.), or latent capacity (Henderson & Fehr), these researchers are building models that can finally match their effort to the difficulty of the task at hand.