Introduction
The Linear Representation Hypothesis (LRH) is a dominant framework in mechanistic interpretability, suggesting that neural networks represent high-level concepts as specific directions or vectors in their activation space. A "strong" version of this hypothesis posits that all meaningful features are encoded linearly. This paper challenges that assumption by demonstrating that small Gated Recurrent Units (GRUs) can develop sophisticated, non-linear internal logics—specifically "onion representations"—to solve sequence tasks when constrained by model capacity.
Methods and Experimental Setup
Researchers trained GRUs of varying sizes (hidden dimensions $N$ from 48 to 1024) on a synthetic repeat task. In this task, the model receives a sequence of random tokens (e.g., a b c d), followed by a trigger signal, and must then output that exact sequence. To investigate how information was stored, the authors used three types of causal interventions:
- Linear Interventions: Using Distributed Alignment Search (DAS) to find linear subspaces corresponding to specific tokens.
- Onion Interventions: A novel approach manipulating the hidden state by adding or subtracting token embeddings scaled by positional magnitudes.
- Probing: Comparing standard linear probes against "autoregressive probes" that simulate the model's own generation process.
The Onion Representation
Analysis revealed that small models ($N \le 64$) do not assign unique directions to each position in a sequence. Instead, they store tokens in the same subspace but at different orders of magnitude. This creates a layered structure resembling an onion, where the most recent or relevant token occupies the outermost layer with the largest magnitude.
Retrieval from this representation involves a process of destructive reading. As the model generates each token, it subtracts that token's representation from its hidden state. This "peels" the onion, revealing the next token at the subsequent magnitude level. This mechanism allows the model to compress a full sequence into a single hidden state without requiring a separate linear dimension for every position.
Key Results and Phase Transitions
The study found a clear phase transition based on model capacity. Large models ($N \ge 256$) have sufficient dimensionality to follow the Linear Representation Hypothesis, storing tokens in distinct linear subspaces. For these models, linear unigram interventions were 100% successful.
In contrast, small models ($N = 64$) opted for the compressed onion strategy. For these models:
- Linear interventions failed completely (0% success rate).
- Onion interventions were highly successful (87% accuracy).
- Linear probes failed to decode the hidden state, while autoregressive "Onion Probes" performed almost perfectly.
Numerical precision tests showed that 16-bit precision (bf16) was sufficient to maintain these magnitude-based layers, though a physical limit likely exists before noise overwhelms the signal.
Implications for Interpretability
These findings identify a significant blind spot for standard interpretability tools. If researchers only look for linear features, they may fundamentally misunderstand the mechanics of smaller or more compressed models. While it is currently unclear if onion representations appear in large-scale LLMs, the fact that these models operate in a relatively small residual stream compared to their total feature space suggests that similar compression pressures—and non-linear solutions—might exist. This work underscores the need for interpretability methods that can detect and explain non-linear representational strategies.
Summary updated