Researchers at EleutherAI have explored a novel frontier in mechanistic interpretability: the "Microscope AI" vision of replacing a neural network’s opaque mathematical weights with human-readable natural language without compromising the model's functionality. This study investigates whether we can "run" a model using text-based explanations as a direct substitute for original weights.
The Problem: Beyond Polysemanticity
Large Language Models (LLMs) are often viewed as "black boxes." While mechanistic interpretability aims to decode them, researchers frequently encounter polysemanticity, where a single neuron activates for multiple unrelated concepts. Tools like Sparse Autoencoders (SAEs) help by decomposing these activations into thousands of "interpretable" features (latents). However, a human-readable explanation (e.g., "this feature activates on legal terminology") may not capture the full computational nuance required for the model to function. This paper tests if automated natural language explanations can faithfully replicate the underlying mathematical operations.
Methodology: The Rewriting Pipeline
The authors developed a multi-step pipeline to test this concept on the Pythia 160M model, specifically targeting the MLP in the 6th layer.
- Transcoders: Instead of standard SAEs, the team used "transcoders" trained to predict the output of a feedforward (FFN) layer given its input, acting as a pluggable replacement for the original MLP.
- Automated Explanations: An automated pipeline generated one-sentence natural language explanations for each of the 32,768 latents in the transcoder.
- The Simulator: The encoder part of the transcoder was replaced with a second LLM (Llama 3 Instruct 8B). This simulator predicts feature activation levels based on the text explanation and the current context.
- Quantile Normalization: Because raw LLM predictions are poorly calibrated, the authors used quantile normalization to map predicted values onto the actual distribution of the original model's activations, ensuring signal consistency.
Key Results and the "Specificity Gap"
The experiments revealed a significant "specificity gap." Current automated explanations are often too broad; for instance, a simulator might activate a feature for every "verb" when the original model only cares about "past-tense verbs in legal contexts." This lack of precision leads to high false-positive rates.
As shown in the performance metrics, replacing features with simulated versions based on text explanations currently performs similarly to "zero-ablation" (simply turning the features off). This suggests that current natural language descriptions do not yet provide enough computational signal to maintain model performance. When all features in an MLP layer were replaced, the model's cross-entropy loss rose significantly, with performance dropping to levels comparable to a model trained on only 10-15% of its data.
Calibration and Interpretability
Calibration proved essential; without quantile normalization, the rewritten model's performance collapsed entirely. Interestingly, the study found a correlation between interpretability and simulation accuracy: features that scored higher on automated interpretability metrics were easier for the LLM to simulate accurately, suggesting a link between human-understandability and computational faithfulness.
Conclusion and Future Directions
While the technical pipeline for rewriting a model in natural language is functional, the results serve as a reality check. One-sentence explanations are insufficient for complex model behavior. Future work may require more detailed "contrastive explanations" to improve specificity. Ultimately, while we can describe what a model is doing in English, we are still a long way from rebuilding a high-performing model entirely from those descriptions.
Summary updated