TL;DR
DECODE is a visual analytics system that reintroduces human oversight into the automated process of interpreting Large Language Model (LLM) internals. By using a multi-stage diagnostic pipeline and an active learning workflow, it allows researchers to efficiently identify whether interpretability failures stem from the underlying model features or the automated explanation process itself.
What Problem Does This Paper Address?
As Large Language Models (LLMs) grow more complex, researchers have turned to mechanistic interpretability to understand their inner workings. A popular technique involves using Sparse Autoencoders (SAEs) to decompose dense neural activations into thousands of "interpretable" latent features.
However, the sheer scale of these features (often hundreds of thousands) makes manual inspection impossible. To cope, researchers use a "self-referential" automated loop: one LLM generates an explanation for a feature, and another LLM scores how "faithful" that explanation is.
This creates a major reliability gap:
- No Ground Truth: There is no objective reference to validate if an LLM’s explanation is actually correct.
- The "Homework Grading" Problem: If an interpretability score is low, is it because the SAE extracted a "noisy" (meaningless) feature, or because the LLM explainer did a poor job?
- Scalability: Human experts can't audit enough features to provide a statistically significant check on the automated process.
Methods
The authors present DECODE (Diagnosing Explanations via COnsensus-Driven Evaluation), which tackles these issues through three core innovations:
1. A Three-Stage Diagnostic Pipeline
Rather than asking a human to judge everything at once, DECODE breaks the task into sequential steps (see Figure 1):
- Stage 1: Structural Soundness. Identifies "Incoherent Splitting," where the SAE has arbitrarily split a single concept into two indistinguishable features.
- Stage 2: Explanation Adequacy. Determines if the generated explanations faithfully describe the feature's activation pattern.
- Stage 3: Failure Attribution. For features that fail, it attributes the cause to either the explainer (e.g., "Missed Syntax" or "Missed Context") or the SAE itself ("Noisy Activation").
2. Consensus as a Proxy for Truth
Since there is no ground truth, DECODE uses inter-coder consensus. It employs three different LLM explainers (Llama 3.1, GPT-4o mini, and Gemini 2.5) and three distinct interpretability metrics. If multiple independent models agree on a pattern, it is treated as a more reliable signal.
3. Three-Phase Active Seeding
To scale human judgment, DECODE uses a semi-automatic labeling workflow (Figure 2). A classifier (SVM) learns from a small set of human-labeled "seeds" to categorize the rest of the features. The system strategically picks which features a human should label next using three phases:
- Prototype-first: Labeling the most representative features.
- Uncertainty-first: Labeling features near the classifier's decision boundary.
- Disagreement-first: Labeling features where a committee of auxiliary classifiers disagrees with the main SVM.
Datasets
The authors evaluated DECODE using:
- Model: Gemma-2-9b.
- Features: 16,384 features extracted from the residual stream at layer 31 using a Sparse Autoencoder.
- Evidence: 20 activating examples per feature sourced from Neuronpedia, providing token-level activation strengths.
Key Insights and Ideas
- The SAE is often the bottleneck: In their case study, experts found that "Noisy Activation" (an SAE-driven failure) was the dominant failure mode. This suggests that improving the feature extraction process (the SAE) might be more impactful than simply using better LLMs for explanation.
- Pairwise comparison is powerful: Some failures, like "Incoherent Splitting," are invisible when looking at a single feature. Only by juxtaposing two similar features (as shown in Figure 3) can a human see that the boundary between them is arbitrary.
- Beware of Anchoring Bias: The study found that if experts saw the LLM-generated consensus before looking at the raw data, they were more likely to be biased toward the LLM's interpretation. The authors recommend an "evidence-first" workflow.
Key Results
- Efficiency: Three domain experts were able to diagnose the entire 16,384-feature set in roughly two hours each by manually labeling only ~100-150 units.
- Accuracy: Aggregating judgments from the three experts yielded a macro-F1 score of 0.62 on a reference set, significantly outperforming a "score-only" baseline (0.36) that relied purely on automated metrics.
- Diagnostic Breakdown: The experts identified that roughly 45-51% of the features were "Well-Explained," while the rest suffered from various structural or explanation-level failures.
Limitations and Future Work
- Label Propagation Quality: While the system scales well, the "propagated" labels (those assigned by the classifier) are less accurate than manual ones (0.23–0.29 F1).
- Class Imbalance: Rare failure types (like "Missed Syntax") are harder for the active learning classifier to pick up because they appear so infrequently in the seed set.
- Scope: The current framework focuses on token-level activations. Future work could extend this to "polysemantic" features (features that represent multiple concepts) or output-based feature interpretations.
- Human-in-the-loop Fatigue: Even with active seeding, Stage 3 (Failure Attribution) remains cognitively demanding for experts. The authors suggest further decomposing this stage into simpler binary decisions.
Summary updated