In the study of neural networks, we often treat individual neurons as the fundamental units of meaning. We visualize their "tuning curves" or assign them labels like "the dog neuron." However, from a linear algebra perspective, this focus is somewhat arbitrary. If you take the activation vector of a hidden layer and apply a random orthogonal rotation, the information content remains identical, and the network’s final output is unchanged.
This raises a fundamental question in mechanistic interpretability: Is the specific coordinate system (the "basis") chosen by a neural network meaningful, or are we just looking at one of infinitely many equivalent rotations?
The Rotational Invariance Paradox
Mathematically, many operations in a neural network are rotationally invariant. If a layer consists only of linear transformations, any rotation of the input can be compensated for by an inverse rotation of the weights. However, as Bolei Zhou, David Bau, and colleagues demonstrated in Interpreting Deep Visual Representations via Network Dissection, neural networks naturally converge to a representation where individual units align with human-understandable concepts.
Crucially, they found that if you randomly rotate this activation space, the model's accuracy stays the same, but the semantic clarity of the neurons vanishes. This suggests that the "neuron basis" is privileged—it is not just one of many equivalent frames of reference, but a specific orientation that the training process prefers.
Why Symmetries Break
Why does a network prefer one orientation over another? The answer lies in the nonlinearities. Charles Godfrey, Henry Kvinge, and their team introduced the concept of intertwiner groups in On the Symmetries of Deep Learning Models and their Internal Representations to explain this. They argue that the specific nonlinear functions used in a model (like ReLU) act as constraints that propagate symmetry—or the lack thereof—from the weights to the internal representations.
Davis Brown and Yamini Bansal provided empirical weight to this in On Privileged and Convergent Bases in Neural Network Representations. They showed that if you rotate the activations of one layer, the subsequent layers cannot easily recover the lost performance. This indicates that the "downstream" parts of the network are "expecting" information to arrive along specific coordinate axes. The individual neuron directions carry unique functional significance that linear transformations cannot simply remap.
Distributed Representations and Subspace Alignment
While the neuron basis is privileged, it is rarely perfect. We often encounter polysemanticity, where a single neuron responds to multiple unrelated concepts (e.g., "car wheels" and "cat paws"). Laura O'Mahony and Mara Graziani highlight in Disentangling Neuron Representations with Concept Vectors that semantic meaning is often distributed across high-dimensional space rather than being perfectly "one neuron, one concept."
To find these hidden meanings, researchers have moved toward Distributed Alignment Search (DAS). In Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations, Atticus Geiger, Zhengxuan Wu, and Christopher Potts propose that instead of looking at neurons, we should look for interpretable directions.
DAS works by learning a rotation matrix that aligns a specific subspace of neural activations with a target causal variable (like a boolean logic gate). This allows researchers to "rotate" the model's internal state into a frame where the underlying algorithm becomes visible. As shown in Interpretability at Scale: Identifying Causal Mechanisms in Alpaca, this method can reveal structured symbolic logic in massive models like Alpaca that standard neuron-by-neuron analysis would miss.
Toward a New Basis
The tension between the "privileged neuron basis" and the "distributed concept direction" is currently being reconciled through methods like Sparse Autoencoders (SAEs). In Sparse Autoencoders Find Highly Interpretable Features in Language Models, Robert Huben and colleagues argue that the model's "native" basis is often an overcomplete set of directions. By training an autoencoder with a sparsity penalty on model activations, they can "untangle" polysemantic neurons into a new basis of monosemantic features.
This suggests that while the network's architecture creates a privileged basis, the true interpretable units are often directions within that space. Whether through learned rotations in RAVEL or convex optimization in Convex Efficient Coding, the goal of modern interpretability is to find the specific geometric frame where the model's "distributed" mess becomes a "privileged" set of clear, causal concepts.