Standard Transformer architectures are notoriously opaque. While their mathematical foundations—composed of matrix multiplications and non-linear activations—are straightforward, their internal representations are highly complex. Facts, reasoning steps, and semantic structures are typically distributed across millions of parameters, making it difficult to trace why a model makes a specific prediction.
To bridge this interpretability gap, researchers are developing frameworks to unpack the internal mechanics of Transformers. By analyzing token geometry, reverse-engineering internal computational circuits, squeezing information flow, and localizing knowledge storage, we can begin to understand the structured algorithms emergent within these black boxes.
The Geometric Evolution of Token Representations
To understand how a Transformer processes information, it helps to look at the geometry of its hidden states. As tokens pass through successive layers, they do not move randomly; instead, they form distinct trajectories in the high-dimensional representation space.
By modeling tokens mathematically as particles moving on a sphere, Giuseppe Bruno et al. demonstrated in Emergence of meta-stable clustering in mean-field transformer models that tokens linger in "meta-stable" clusters before eventually collapsing into a consensus state. This clustering phase explains how finite-depth models preserve diverse, meaningful semantic groupings across layers.
However, standard self-attention relies on empirical heuristics like the softmax activation, which complicates formal mathematical analysis. To make these internal groupings inherently interpretable, researchers are designing "white-box" architectures derived from first principles.
For instance, Ziyang Wu et al. introduced the Token Statistics Transformer (Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction), which replaces traditional attention with a mechanism based on Maximal Coding Rate Reduction. This framework provides an explicit mathematical rule for token clustering, making the model's internal grouping logic transparent by design.
Similarly, Peng Wang et al. proposed the Attention-only Transformer (Attention-Only Transformers via Unrolled Subspace Denoising), demonstrating that stripping away non-linear Multi-Layer Perceptrons (MLPs) allows each layer to be interpreted as an explicit step in subspace denoising. This formulation mathematically guarantees that each layer progressively improves the signal-to-noise ratio of the data representations.
Reverse-Engineering Internal Circuits
Rather than viewing a Transformer as a monolithic weight matrix, mechanistic interpretability treats the network as a structured computer program composed of discrete, functional sub-networks called circuits.
A foundational example of this approach was provided by Kevin Ro Wang et al. in Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. The authors successfully mapped a precise circuit of 28 attention heads that collectively perform indirect object identification, proving that large language models execute structured, human-understandable algorithms to solve linguistic tasks.
To scale this analysis to more complex reasoning, Di Wang et al. developed the Self-Influence Circuit Analysis Framework (SICAF) in Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning. By tracking how layers influence themselves and each other, the framework maps the model's step-by-step progression from initial entity identification to final logical synthesis.
This modularity extends to vision tasks as well. In Discovering Influential Neuron Path in Vision Transformers, Yifan Wang et al. utilized a Joint Attribution Score to isolate sparse sequences of neurons across layers. They discovered that Vision Transformers rely on highly localized paths that naturally cluster by semantic category, allowing for efficient model pruning without sacrificing accuracy.
[Input Tokens] ---> [Entity Identification Circuit] ---> [Logical Synthesis Circuit] ---> [Output Prediction]
Squeezing and Probing the Information Flow
Another way to build intuition about a Transformer's decision-making process is to audit the information flow during a forward pass. How do we determine which tokens or hidden states are truly driving a prediction?
One powerful technique is to force the model through an informational "squeeze." Zhiying Jiang et al. applied this concept in Inserting Information Bottleneck for Attribution in Transformers by injecting noise into specific hidden layers. By minimizing the information retained while maintaining prediction accuracy, the bottleneck forces the model to discard non-essential data, isolating the exact input tokens responsible for the decision.
For broader conceptual explanations, Fanny Jourdan et al. developed COCKATIEL (COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP). This post-hoc framework uses Non-Negative Matrix Factorization to cluster internal activations into high-level, human-interpretable concepts, ranking them by their global sensitivity.
Taking a causal approach, Raanan Y. Rohekar et al. demonstrated in Causal Interpretation of Self-Attention in Pre-Trained Transformers that the self-attention mechanism can be mathematically interpreted as estimating a Structural Equation Model. Their method, CLEANN, leverages this insight to extract minimal causal graphs that explain model predictions.
Localizing and Editing Distributed Knowledge
In standard architectures, factual knowledge is smeared across weights as statistical approximations, making it difficult to update or erase specific facts without retraining. A major focus of structural interpretability is localizing where these concepts live and designing mechanisms to isolate them.
In the domain of generative vision models, Arman Zarei et al. investigated the organizational structure of Diffusion Transformers (DiTs) in Localizing Knowledge in Diffusion Transformers. The authors pinpointed specific blocks responsible for storing distinct concepts, such as artistic styles or copyrighted characters, enabling targeted model editing and unlearning with minimal disruption to the model's broader capabilities. Similarly, in video generation, Siyoon Jin et al. identified specific "interaction-dominant layers" in MATRIX: Mask Track Alignment for Interaction-aware Video Generation that govern how subjects and objects interact, allowing for precise grounding interventions.
To avoid the problem of distributed weight entanglement altogether, researchers are exploring alternative, modular architectures:
- Explicit Memory Slots: Damai Dai et al. proposed the Neural Knowledge Bank (Neural Knowledge Bank for Pretrained Transformers), a modular plug-in that introduces dedicated memory slots into pretrained models, allowing facts to be updated directly without altering the base network's general language capabilities.
- Linear Representation Aggregation: To enforce interpretability at the word level, Hao Sun evaluated the Backpack architecture in Character-level Chinese Backpack Language Models. The Backpack model ensures that contextual token predictions are formed as strict linear combinations of predictable, non-contextual word representations, making the influence of individual weights transparent and editable.