DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process

Dohyun Kim, Jaeung Lee, Donghee Hong, Jaemin JoarXiv 2026 · arXiv

TL;DR

DECODE is a visual analytics system that reintroduces human oversight into the automated process of interpreting Large Language Model (LLM) internals. By using a multi-stage diagnostic pipeline and an active learning workflow, it allows researchers to efficiently identify whether interpretability failures stem from the underlying model features or the automated explanation process itself.

What Problem Does This Paper Address?

As Large Language Models (LLMs) grow more complex, researchers have turned to mechanistic interpretability to understand their inner workings. A popular technique involves using Sparse Autoencoders (SAEs) to decompose dense neural activations into thousands of "interpretable" latent features.

However, the sheer scale of these features (often hundreds of thousands) makes manual inspection impossible. To cope, researchers use a "self-referential" automated loop: one LLM generates an explanation for a feature, and another LLM scores how "faithful" that explanation is.

This creates a major reliability gap:

  1. No Ground Truth: There is no objective reference to validate if an LLM’s explanation is actually correct.
  2. The "Homework Grading" Problem: If an interpretability score is low, is it because the SAE extracted a "noisy" (meaningless) feature, or because the LLM explainer did a poor job?
  3. Scalability: Human experts can't audit enough features to provide a statistically significant check on the automated process.

Methods

The authors present DECODE (Diagnosing Explanations via COnsensus-Driven Evaluation), which tackles these issues through three core innovations:

1. A Three-Stage Diagnostic Pipeline

Rather than asking a human to judge everything at once, DECODE breaks the task into sequential steps (see Figure 1):

  • Stage 1: Structural Soundness. Identifies "Incoherent Splitting," where the SAE has arbitrarily split a single concept into two indistinguishable features.
  • Stage 2: Explanation Adequacy. Determines if the generated explanations faithfully describe the feature's activation pattern.
  • Stage 3: Failure Attribution. For features that fail, it attributes the cause to either the explainer (e.g., "Missed Syntax" or "Missed Context") or the SAE itself ("Noisy Activation").
Figure 1: Overview of DECODE. (A) Automated interpretation process extracts SAE features, generates explanations with three explainers, and evaluates them with three metrics. The red dashed border highlights the self-referential loop. (B) Three-stage diagnostic pipeline sequentially assigns features into five diagnostic categories, passing unresolved features to the next stage via intermediate categories. (C) Three-phase active seeding protocol recommends features based on representativeness, decision boundary proximity, and classifier ensemble disagreement.

2. Consensus as a Proxy for Truth

Since there is no ground truth, DECODE uses inter-coder consensus. It employs three different LLM explainers (Llama 3.1, GPT-4o mini, and Gemini 2.5) and three distinct interpretability metrics. If multiple independent models agree on a pattern, it is treated as a more reliable signal.

3. Three-Phase Active Seeding

To scale human judgment, DECODE uses a semi-automatic labeling workflow (Figure 2). A classifier (SVM) learns from a small set of human-labeled "seeds" to categorize the rest of the features. The system strategically picks which features a human should label next using three phases:

  1. Prototype-first: Labeling the most representative features.
  2. Uncertainty-first: Labeling features near the classifier's decision boundary.
  3. Disagreement-first: Labeling features where a committee of auxiliary classifiers disagrees with the main SVM.

Datasets

The authors evaluated DECODE using:

  • Model: Gemma-2-9b.
  • Features: 16,384 features extracted from the residual stream at layer 31 using a Sparse Autoencoder.
  • Evidence: 20 activating examples per feature sourced from Neuronpedia, providing token-level activation strengths.

Key Insights and Ideas

  • The SAE is often the bottleneck: In their case study, experts found that "Noisy Activation" (an SAE-driven failure) was the dominant failure mode. This suggests that improving the feature extraction process (the SAE) might be more impactful than simply using better LLMs for explanation.
  • Pairwise comparison is powerful: Some failures, like "Incoherent Splitting," are invisible when looking at a single feature. Only by juxtaposing two similar features (as shown in Figure 3) can a human see that the boundary between them is arbitrary.
  • Beware of Anchoring Bias: The study found that if experts saw the LLM-generated consensus before looking at the raw data, they were more likely to be biased toward the LLM's interpretation. The authors recommend an "evidence-first" workflow.
Figure 3: Stage 1 interface. (A) Stage tabs indicate diagnostic progress (Stage 1 active). (B) Overview Panel displays a decoder similarity histogram. A draggable threshold at 0.40 separates inspection candidates (above) from features assigned Coherent Separation (below). (C) Workspace Panel: (c1) Seeding List ranks candidate pairs, showing 20 representatives in prototype-first phase; (c2) Inspection View juxtaposes feature 13143 and 13256 both activating on tokens starting with “sta”; (c3) Labeling View displays confidence histogram and stability chart after three training iterations.

Key Results

  • Efficiency: Three domain experts were able to diagnose the entire 16,384-feature set in roughly two hours each by manually labeling only ~100-150 units.
  • Accuracy: Aggregating judgments from the three experts yielded a macro-F1 score of 0.62 on a reference set, significantly outperforming a "score-only" baseline (0.36) that relied purely on automated metrics.
  • Diagnostic Breakdown: The experts identified that roughly 45-51% of the features were "Well-Explained," while the rest suffered from various structural or explanation-level failures.
Table 1: Terminal diagnostic categories used in DECODE.

Limitations and Future Work

  • Label Propagation Quality: While the system scales well, the "propagated" labels (those assigned by the classifier) are less accurate than manual ones (0.23–0.29 F1).
  • Class Imbalance: Rare failure types (like "Missed Syntax") are harder for the active learning classifier to pick up because they appear so infrequently in the seed set.
  • Scope: The current framework focuses on token-level activations. Future work could extend this to "polysemantic" features (features that represent multiple concepts) or output-based feature interpretations.
  • Human-in-the-loop Fatigue: Even with active seeding, Stage 3 (Failure Attribution) remains cognitively demanding for experts. The authors suggest further decomposing this stage into simpler binary decisions.

Summary updated

Abstract

Tools for interpreting the internal representations of Large Language Models (LLMs) increasingly rely on LLMs themselves to both generate and evaluate explanations, creating a self-referential automated interpretation process with minimal human oversight. This automation raises significant reliability concerns, which are exacerbated by the absence of ground truth. In practice, it is difficult to validate explanations or to attribute interpretability failures to their source -- whether they originate from the tool's extracted representations or from the LLM used in the process. To address these challenges, we present DECODE, a visual analytics system that reintroduces human oversight into the automated interpretation process. Our approach decomposes the diagnostic workflow into three progressive stages, isolating failure sources ranging from structural defects to explanation-level misalignments. Drawing inspiration from inter-coder consensus in qualitative analysis, we utilize consensus among independently generated explanations and their evaluation metrics as a proxy for reliability. To further scale human inspection, we employ a three-phase active seeding protocol that strategically recommends high-priority instances for manual review. Finally, we demonstrate the effectiveness of our approach through a case study on 16,384 Sparse Autoencoder features, showing that it can efficiently identify and attribute interpretability failures at scale.
BibTeX
@misc{kim2026decode,
  author = {Dohyun Kim and Jaeung Lee and Donghee Hong and Jaemin Jo},
  title = {{DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process}},
  year = {2026},
  eprint = {2611.17856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17856},
}
Figure