TL;DR
This paper reveals that Transformers execute multi-step reasoning via a buffer mechanism, where intermediate conclusions are stored in distinct subspaces defined by value-output projection matrices ($W^{vo}$) and retrieved by query-key attention heads ($W^{qk}$). This perspective explains why Chain-of-Thought (CoT) is so effective—it allows models to repeatedly reuse a small set of buffers across generated tokens rather than burning through network layers. Exploiting this insight, the authors introduce a Random Matrix-Based Algorithm (RMBA) that accelerates generalization on the PrOntoQA benchmark by 75%.
What Problem Does This Paper Address?
While Large Language Models (LLMs) solve complex problems when prompted with step-by-step scratchpads like Chain-of-Thought (horizontal thinking), their ability to solve multi-step logic in a single forward pass (vertical thinking) degrades rapidly—often collapsing after just three steps.
Prior mechanistic interpretability work identified single-step primitives (such as induction heads), but multi-step reasoning is not a simple linear cascade of single steps. How Transformers store, isolate, and route intermediate logical deductions across layers without causing cross-talk has remained an open question. Understanding this internal circuit is essential for designing architectures and initialization strategies that learn logical deduction efficiently.
Methods
- The Buffer Mechanism: The authors formalize the intuition that attention value-output projections ($W^{vo(l)} = W^{v(l)} W^{o(l)}$) act as a collection of nearly orthogonal buffers. Intermediate results are mapped into separate subspaces ($[a]W^{vo(l)}$), while query-key heads ($W^{qk(l)} = W^{q(l)} W^{k(l)\top}$) serve as "extractors." Reasoning occurs when an extractor aligns with a buffer such that their product forms a diagonal kernel ($Ker^{(l)} \approx I$), triggering "same-token matching" to link an intermediate deduction to the premise of the next step.
- Buffer-Level Causal Intervention: Rather than perturbing entire token representations, the authors surgically swapped values within individual buffer subspaces across layers to confirm where intermediate reasoning states reside.
- Analytical Weight Construction: Using the buffer formulation, the authors proved and demonstrated that an $L$-layer Transformer can execute $(L-1)$-step reasoning zero-shot without any training, merely by setting $W^{qk}$ and $W^{vo}$ to predetermined permutation and random matrices.
- Random Matrix-Based Algorithm (RMBA): To improve data efficiency during training, prior work proposed adding identity matrices to attention projections (Identity Matrix-Based Algorithm, or IMBA). However, the authors show that in multi-step reasoning, adding identity matrices creates severe informational interference. Instead, RMBA regularizes the weights by adding fixed Gaussian random matrices: $$W^{qk(l)} + \alpha^{(l)} Z^{(l-1)}, \quad W^{vo(l)} + \beta^{(l)} Z^{(l)}$$ where $\alpha, \beta$ are learnable scalars and $Z \sim \mathcal{N}(0, 1/d_m)$. This injects fresh, mutually orthogonal buffer subspaces from the start of training.
Datasets
- Synthetic Symbolic Multi-Step Reasoning Dataset: An abstract benchmark serialized as token pairs representing directed reasoning edges (e.g., $A \to B, B \to C$). The authors created distinct in-distribution (ID) and out-of-distribution (OOD) splits using modular arithmetic splits, forcing the model to perform in-context reasoning over unseen relation pairs.
- PrOntoQA: A 0.5M synthetic question-answering dataset evaluating 3-step first-order logic deduction, formatted as natural language statements concluding in True/False questions.
- LLM Benchmarking: Tested closed and open models (GPT-4, GPT-4o, Claude 3.5 Sonnet/Haiku, and Phi-3) on multi-step reasoning chains.
Key Insights and Ideas
-
Why CoT is Computationally Superior (Buffer Reuse): In a single forward pass (vertical thinking), each step consumes layer depth because the model must allocate a new, orthogonal buffer ($W^{vo(l)}$) to avoid overwriting earlier deductions. In contrast, horizontal thinking (CoT) outputs each intermediate deduction as a new text token. This resets the deduction into the base residual stream buffer ($I$), allowing a tiny 2-layer Transformer to solve arbitrarily long reasoning chains by repeatedly cycling through the exact same two buffers.
-
Why Identity Regularization Fails for Multi-Step: Adding identity matrices ($W^{vo} + \beta I$) causes representation leakage between premises and deductions ($[a]W^{vo} + \beta [a] + [b]$). Adding independent random matrices (RMBA) preserves subspace orthogonality, preventing catastrophic feature interference.
-
Evidence in Production LLMs: Inspecting pre-trained Phi-3 reveals that $W^{qk}$ and $W^{vo}$ matrices in middle layers (layers 5–20) naturally align into diagonal kernel structures, and their flattened $W^{vo}$ matrices are pairwise orthogonal, confirming the buffer mechanism emerges in real-world models.
Key Results
- Sample Efficiency: On PrOntoQA, training a 12-layer GPT-2 small with RMBA reduced the number of iterations required to achieve out-of-distribution generalization by ~75% (generalization began around 0.25 epochs, compared to ~1.0 epoch for baseline GPT-2).
- Distractor Robustness: When random distractor tokens were inserted into test inputs, RMBA retained baseline-level accuracy (>85%), whereas IMBA dropped steeply to ~25–35%.
- Zero-Shot Analytical Construction: An untrained 8-layer Transformer constructed using random buffer weights achieved 100% accuracy on 7-step reasoning tasks across natural, reversed, and scrambled premise orders.
- CoT Scaling: A minimal 2-layer model trained exclusively on 1-step sequences of length 13 generalized to 6-step reasoning chains via autoregressive feedback with >57.6% accuracy.
Limitations and Future Work
- Simplified Assumptions: The analytical derivation relies on the assumption that hidden dimension $d_m$ is sufficiently large for near-perfect random orthogonality, whereas real-world models operate under tight dimension bottlenecks.
- Feedforward Network (FFN) Omission: The core theoretical formulation simplifies attention by abstracting away non-linear feedforward layers, although empirical evaluations included standard Transformer blocks.
- Domain Scope: Experiments primarily focus on formal symbolic chains and synthetic logical deduction (PrOntoQA); verifying whether buffer mechanisms govern fuzzy, semantic, or multi-hop natural language tasks remains for future work.
Summary updated