Modern language models are "black boxes" not just because their weights are hidden, but because their internal representations are fundamentally messy. A single neuron often responds to a confusing mix of unrelated concepts—a problem known as polysemanticity. To make sense of these models, researchers are developing automated pipelines that first disentangle these representations and then use secondary "explainer" models to label and validate what they find.
Disentangling the Feature Soup
The first step in understanding a model's internal state is moving from raw neurons to "features." As Robert Huben, Hoagy Cunningham, and colleagues describe in Sparse Autoencoders Find Highly Interpretable Features in Language Models, Sparse Autoencoders (SAEs) act as a mathematical prism. They take the dense, polysemantic activations of a model and project them into a much larger, "overcomplete" space where each dimension ideally represents a single, monosemantic concept.
By forcing the autoencoder to use only a few active features at a time (sparsity), the model is incentivized to find the most fundamental building blocks of the data. This approach has been extended beyond text to multi-modal models by Kaichen Zhang and Ziwei Liu, who demonstrate that SAEs can isolate visual concepts that are then steerable to reduce hallucinations (Large Multi-modal Models Can Interpret Features in Large Multi-modal Models).
Scaling Interpretation with LLM Proxies
Once we have millions of isolated features, manual inspection becomes impossible. The core innovation of this research direction is using a high-capacity Large Language Model (LLM) as an automated scientist.
In the pipeline proposed by Lee Sharkey and Jeffrey Wu, the process follows a "generate-and-test" loop:
- Generation: An explainer LLM looks at the contexts that most strongly activate a specific feature and synthesizes a natural language description (e.g., "this feature fires on mentions of legal contracts").
- Validation: A separate model (or the same one in a different role) acts as a "proxy user." It is given the explanation and asked to predict whether the feature will fire on new, unseen text.
This "Forward Simulatability" metric, as explored by Roberto Barile in LP-DIXIT, provides a quantitative score for interpretability. If a model can accurately predict a feature's behavior based solely on the explanation, the explanation is considered faithful and useful.
Refining the Language of Interpretability
While natural language is flexible, it can be vague. Angie Boggust and Fred Hohman suggest that we can improve this process by using semantic regexes—a structured language that combines linguistic primitives with compositional modifiers (Semantic Regexes: Auto-Interpreting LLM Features with a Structured Language). This allows the automated pipeline to produce descriptions that are not only human-readable but also computationally precise.
Similarly, researchers like Tomáš Musil have shown that LLMs can replace humans in traditional psychometric tests, such as the "Word Intruder Test," to measure the semantic coherence of embedding dimensions at scale (Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test).
The Sanity Check: Learning vs. Statistics
Despite the promise of these automated scores, the field faces a significant "sanity check" challenge. Thomas Heap and Tim Lawson have pointed out a potential flaw: current automated interpretability metrics often fail to distinguish between a fully trained model and a completely random one (Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers).
This suggests that an LLM might be able to write a "plausible" explanation for a feature simply because the input data has a clear statistical structure (like "words related to biology"), even if the model hasn't actually learned a meaningful circuit for that concept. This tension between plausibility (how much an explanation makes sense to us) and faithfulness (how much it reflects the model's actual computation) remains a central focus for researchers like Xiaolei Lu (Does Faithfulness Conflict with Plausibility?).