For years, mechanistic interpretability has relied on a beautifully simple premise: the Linear Representation Hypothesis (LRH). This hypothesis posits that a neural network stores semantic concepts as distinct, directional vectors or flat subspaces within its high-dimensional activation space.
Under this framework, understanding a model feels like doing basic geometry. If you want to find out if a model knows whether a statement is true, you look for a single "truth" direction (Burns et al.). If you want to make a model less toxic or change its grammatical output, you find the corresponding vector and nudge the activations along that straight line (Ferrando et al.; Patel et al.).
Theoretically, this makes sense as a starting point. As shown by the authors of On the Origins of Linear Representations in Large Language Models, next-token prediction objectives and the implicit bias of gradient descent naturally encourage models to learn linear structures under certain conditions. However, a growing body of research suggests that this linear assumption is merely a shallow approximation. When concepts become complex, abstract, or deeply entangled, the straight lines warp into intricate, non-linear manifolds.
Where Linearity Breaks Down
Why should we expect representations to be non-linear? The answer lies in how deep networks optimize for capacity and complexity.
To pack an immense dictionary of real-world concepts into a limited number of hidden dimensions, networks cannot rely solely on flat, orthogonal directions. Instead, they develop hierarchical, multi-dimensional, and non-linear structures to manage overlapping features (Costa et al.). Furthermore, because Transformers are built out of non-linear activation functions and normalization layers, the direct mapping from internal states to output behaviors is inherently non-linear (Dunefsky et al.).
When we try to force these complex internal structures into a linear box, our interpretability tools begin to fail:
- The Guidability Gap: Dimitri von Rütte demonstrates that while simple concepts like truthfulness have clear linear signatures, complex or subjective traits like humor and creativity exhibit a "guidability gap." Even if a model can internally detect these traits, simple linear vector addition fails to steer them effectively because their underlying generative processes are non-linear.
- Hidden World Models: Internal representations of structured environments—such as the board state in a game of Othello—are often encoded non-linearly. Kenneth Li demonstrates that these internal world models remain completely invisible to simple linear probes, yet they can be readily decoded by more complex, non-linear classifiers.
- Distributed Concepts: Abstract conceptual knowledge (Peng et al.) and nuanced linguistic properties like pragmatic bias (Govindarajan et al.) are distributed across layers in complex patterns. Relying strictly on linear classifiers or one-dimensional subspaces (Arora) yields an incomplete and misleading picture of how the model processes information.
The Failure Mode of Linear Interventions
The assumption of linearity isn't just an academic misstep; it actively limits our ability to control and align models. If a concept resides on a curved manifold, treating it as a straight line leads to unpredictable and harmful side effects during steering or erasure.
Linear Projection vs. Non-Linear Manifold
Linear Vector Shift (Fails) Non-Linear Manifold Tracking
/---> (Warped State) .---*---.
/ / \
*-----/-----> (Target) *-----* *
(Start) (Start) (Target)
Applying a straight vector Following the true geometric
pushes the representation manifold preserves the concept
out of distribution. without destroying model fluency.
Consider what happens when you try to steer a model's safety guardrails. If safety mechanisms were simple linear vectors, increasing the intervention intensity would linearly increase safety. But Siu et al. discovered that in weakly aligned models, this relationship is often non-monotonic. Increasing the steering strength can cause unpredictable behavioral flips because the safety features are embedded in a complex, non-linear manifold.
Similarly, the authors of SOM Directions are Better than One show that refusal behavior is not a single "off switch" direction, but a multi-dimensional manifold. Forcing a single-vector intervention leaves parts of the concept unaddressed, allowing adversarial jailbreaks to slip through the unmapped dimensions of the manifold.
This geometric mismatch makes clean concept erasure incredibly difficult. Saillenfest et al. note that linear erasure techniques fail to fully isolate or transform target attributes when they are deeply entangled non-linearly. In fact, Bo Han proves that completely erasing harmful concepts from these non-linear representation spaces is mathematically impossible using purely linear methods without simultaneously damaging the model's broader reasoning capabilities.
Mapping the Non-Linear Frontier
To move past these limitations, researchers are developing new paradigms that respect the topological complexity of latent spaces.
Non-Linear Probing and Decoders
Instead of restricting probes to linear hyperplanes, researchers are using small, non-linear neural networks as diagnostic tools. Michael Kleinman uses non-linear decoders to track how information evolves during training, successfully extracting "usable information" that is present in the hidden states but completely hidden from linear output layers.
Multi-Directional and Topological Mapping
Rather than searching for a single vector, new methods map the broader geometry of the concept space. For example, the authors of SOM Directions are Better than One utilize Self-Organizing Maps (SOMs) to identify and ablate multiple, distinct directions that collectively form a refusal manifold, achieving far more robust control than single-vector methods.
Concept Concentration
Rather than changing the intervention to fit the manifold, what if we could change the manifold to fit the intervention? Bo Han introduces COncept ConcentrAtion (COCA), a method that uses structured reasoning to actively "squash" messy, non-linear harmful concepts into a neat, isolated linear subspace. By forcing the network to linearize its representation of a specific concept, researchers can cleanly remove it using standard linear projections without degrading the surrounding cognitive architecture.