TL;DR
This paper introduces a unified framework for evaluating concept-based explanations in Large Language Models (LLMs) by formalizing concepts as "virtual neurons." The authors propose automated, reliable metrics for faithfulness (via optimized perturbation) and readability (via semantic coherence), validating them through a rigorous meta-evaluation based on classical measurement theory.
What Problem Does This Paper Address?
As LLMs become more complex, we need to understand what high-level patterns they are learning, not just where they are looking (the goal of traditional attribution methods). Concept-based explanations aim to bridge this gap by identifying human-understandable features within a model's hidden layers.
However, the field faces three major hurdles:
- Lack of Unified Formalization: Different methods (like TCAV or Sparse Autoencoders) define "concepts" differently, making side-by-side comparison impossible.
- Evaluation Cost: Assessing if an explanation is "readable" usually requires expensive human studies, which don't scale.
- Metric Reliability: Existing automated metrics often lack validation; we don't know if a high "faithfulness score" actually means the explanation is accurate.
Methods
The authors propose a three-part framework to standardize evaluation:
- Concept Extraction & Formalization: They define a concept as a "virtual neuron" with an activation function $a(h)$ that maps a hidden representation $h$ to a scalar value. This abstraction covers supervised, unsupervised, and interpretable-by-design methods.
- Automated Evaluation Metrics:
- Faithfulness: Measured by perturbing the hidden space. Instead of simple ablation, they use an optimization problem to find the minimal perturbation that zeros out a concept (ablation) or adds to it (gradient), then measure the resulting change in model output.
- Readability: Inspired by topic modeling, they measure the semantic coherence of tokens that maximally activate a concept. They use embedding-based similarities (like BERT embeddings) to see how closely related these tokens are.
- Meta-Evaluation: Borrowing from psychometrics, they evaluate the metrics themselves using Reliability (consistency across re-tests and data subsets) and Validity (how well the metric correlates with human judgment).
Datasets
- The Pile: A massive, diverse dataset used to extract a wide array of concepts.
- Models: The authors primarily analyzed Pythia-70M and GPT-2 (small) to ensure findings were consistent across different architectures.
- Human Ground Truth: A user study with three human labelers scoring 200 concepts provided the "gold standard" for validating the automated readability metrics.
Key Insights and Ideas
- The Reliability Filter: A major "aha moment" was discovering that several popular metrics, including some LLM-based scoring methods, fail basic reliability tests. If a metric gives wildly different scores on two different subsets of the same data, it can't be trusted.
- Input vs. Output Divergence: The patterns that trigger a concept (input side) and the patterns the concept predicts (output side) are often distinct. The paper shows that readability must be measured on both sides independently.
- EmbCos as a Scalable Proxy: The authors found that Embedding Cosine Similarity (EmbCos) is a highly effective, low-cost substitute for human evaluation, outperforming even expensive GPT-4 based evaluation pipelines in terms of correlation with human scores.
Key Results
- Reliability: Several metrics were discarded after failing to meet the 0.9 threshold for subset consistency, including
LLM-ScoreandGRAD-Loss. - Validity:
IN-EmbCosandOUT-EmbCosemerged as top performers for readability, showing strong correlations (Kendall’s $\tau$ up to 0.67) with human ratings. - Baseline Comparison: When comparing explanation methods, supervised methods (TCAV) generally produced more faithful and readable concepts than unsupervised methods (Neurons, SAEs), though Sparse Autoencoders (SAEs) significantly outperformed raw neuron-based explanations.
Limitations and Future Work
- Scope of Evaluation: The framework focuses on faithfulness and readability but does not yet account for robustness (how explanations change under adversarial noise) or stability.
- Model Scale: Due to compute constraints, the study was limited to smaller LLMs (70M-124M parameters). Future work is needed to see if these evaluation patterns hold for frontier models like GPT-4 or Llama-3.
- Readability Nuance: While semantic coherence is a strong proxy, it may not capture the full "meaningfulness" of a concept that a human expert might perceive.
Summary updated