In a standard Transformer, the amount of computation a model can perform is strictly tied to the number of tokens in its input. Each token provides a fixed-size hidden vector that passes through a fixed number of layers. If a problem is computationally "dense"—requiring many operations but described by only a few words—the model may run out of "room" to think before it has to produce an answer.
Recent research has converged on a powerful solution: filler tokens. By inserting semantically empty tokens (like <pause>, ..., or padding) into the sequence, we can decouple a model's computational capacity from the length of the actual input.
The Parallel Workspace
The intuition behind filler tokens is that they act as a parallel scratchpad. In a typical forward pass, each token position's hidden state is updated by attending to other positions. When we add filler tokens, we aren't just adding length; we are adding "computational width."
Sachin Goyal and colleagues demonstrated this in Think before you speak: Training Language Models With Pause Tokens. By training models with a learnable <pause> token, they allowed the Transformer to delay its output. During these pause steps, the self-attention mechanism can move data across a larger set of hidden vectors, refining internal representations before committing to a prediction. This effectively increases the number of parallel operations the model can perform per layer.
Bypassing Architectural Bottlenecks
Filler tokens do more than just add "thinking time"; they solve fundamental structural limitations of the Transformer architecture, particularly in decoder-only models.
In a decoder-only model, causal masking prevents a token from "looking ahead" at future information. Chris Hays and his co-authors explored this in The Impossibility of Inverse Permutation Learning in Transformer Models. They proved that certain tasks, like inverting a permutation, are mathematically impossible for standard decoders because the necessary information flow is blocked. However, by appending "scratch tokens" to the end of the input, the model gains a buffer where it can aggregate information from the entire preceding sequence and perform the required transformations.
Mechanistically, the model treats these tokens as temporary storage. Jannik Brinkmann and his team performed a mechanistic analysis of Transformers trained on symbolic reasoning. They found that models naturally repurpose neutral tokens—like punctuation or delimiters—as "register tokens." These positions act as a parallel working memory, allowing the model to solve sub-problems independently before aggregating the results at the final output token.
Formal Expressivity and Complexity
The impact of these tokens isn't just empirical; it is theoretically transformative. Without filler tokens, a constant-depth Transformer is limited in the types of logic it can process.
Charles London and Varun Kanade provided formal proof in Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers that filler tokens allow shallow architectures to solve complex logic problems that would otherwise require much deeper models. By distributing computation across the sequence length rather than just through layers, the model's effective width expands.
This line of reasoning is extended by William Merrill and Ashish Sabharwal. In Exact Expressive Power of Transformers with Padding, they show that providing a polynomial number of padding tokens relative to the input length allows Transformers to evaluate complex logic formulas in a single pass. This effectively links "padded" Transformers to classic circuit complexity classes, proving that the context window can function as a high-capacity parallel workspace.
Why It Works: Precision and Looping
The power of filler tokens is remarkably robust. Anej Svete and colleagues found in Revisiting Padded Transformer Expressivity that while architectural details like attention type matter less than one might expect, the true drivers of performance in padded models are numeric precision and looping depth.
By decoupling computational capacity from the semantic length of the input, filler tokens allow the model to leverage its full internal precision across a wider "canvas." Whether they are called pause tokens, scratch tokens, or padding, these semantically empty slots provide the necessary "room" for the complex, parallelized data transformations that modern reasoning tasks demand.