Partially Rewriting a Transformer in Natural Language

Gonçalo Paulo, Nora BelrosearXiv 2025 · DOI · arXiv

Researchers at EleutherAI have explored a novel frontier in mechanistic interpretability: the "Microscope AI" vision of replacing a neural network’s opaque mathematical weights with human-readable natural language without compromising the model's functionality. This study investigates whether we can "run" a model using text-based explanations as a direct substitute for original weights.

The Problem: Beyond Polysemanticity

Large Language Models (LLMs) are often viewed as "black boxes." While mechanistic interpretability aims to decode them, researchers frequently encounter polysemanticity, where a single neuron activates for multiple unrelated concepts. Tools like Sparse Autoencoders (SAEs) help by decomposing these activations into thousands of "interpretable" features (latents). However, a human-readable explanation (e.g., "this feature activates on legal terminology") may not capture the full computational nuance required for the model to function. This paper tests if automated natural language explanations can faithfully replicate the underlying mathematical operations.

Methodology: The Rewriting Pipeline

The authors developed a multi-step pipeline to test this concept on the Pythia 160M model, specifically targeting the MLP in the 6th layer.

The three-step pipeline for rewriting model components: generating explanations, predicting activations via an LLM simulator, and substituting model parts with reconstructed vectors.
  1. Transcoders: Instead of standard SAEs, the team used "transcoders" trained to predict the output of a feedforward (FFN) layer given its input, acting as a pluggable replacement for the original MLP.
  2. Automated Explanations: An automated pipeline generated one-sentence natural language explanations for each of the 32,768 latents in the transcoder.
  3. The Simulator: The encoder part of the transcoder was replaced with a second LLM (Llama 3 Instruct 8B). This simulator predicts feature activation levels based on the text explanation and the current context.
  4. Quantile Normalization: Because raw LLM predictions are poorly calibrated, the authors used quantile normalization to map predicted values onto the actual distribution of the original model's activations, ensuring signal consistency.

Key Results and the "Specificity Gap"

The experiments revealed a significant "specificity gap." Current automated explanations are often too broad; for instance, a simulator might activate a feature for every "verb" when the original model only cares about "past-tense verbs in legal contexts." This lack of precision leads to high false-positive rates.

Absolute cross-entropy difference as a function of the fraction of latents substituted, comparing top-scoring features, sampling, and zeroing out for both Transcoders and SAEs.

As shown in the performance metrics, replacing features with simulated versions based on text explanations currently performs similarly to "zero-ablation" (simply turning the features off). This suggests that current natural language descriptions do not yet provide enough computational signal to maintain model performance. When all features in an MLP layer were replaced, the model's cross-entropy loss rose significantly, with performance dropping to levels comparable to a model trained on only 10-15% of its data.

Calibration and Interpretability

Calibration proved essential; without quantile normalization, the rewritten model's performance collapsed entirely. Interestingly, the study found a correlation between interpretability and simulation accuracy: features that scored higher on automated interpretability metrics were easier for the LLM to simulate accurately, suggesting a link between human-understandability and computational faithfulness.

Conclusion and Future Directions

While the technical pipeline for rewriting a model in natural language is functional, the results serve as a reality check. One-sentence explanations are insufficient for complex model behavior. Future work may require more detailed "contrastive explanations" to improve specificity. Ultimately, while we can describe what a model is doing in English, we are still a long way from rebuilding a high-performing model entirely from those descriptions.

Summary updated

Abstract

The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model's final output. With our pipeline, the model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline.
BibTeX
@misc{paulo2025partially,
  author = {Gonçalo Paulo and Nora Belrose},
  title = {{Partially Rewriting a Transformer in Natural Language}},
  year = {2025},
  eprint = {2501.18838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.18838},
}
Figure