Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability - The Strange Science Part I.ii

Kola Ayonrinde, Louis JaburiarXiv 2025 · DOI · arXiv

Introduction: The Crisis of Interpretability

Mechanistic Interpretability (MI) aims to reverse-engineer neural networks into human-understandable components. However, the field currently lacks a universal "yardstick" for deciding which explanations are actually good. Researchers often encounter "Interpretability Illusions"—explanations that seem plausible to humans but fail to reflect the model's true internal logic. Without objective criteria, the community risks producing a collection of "just-so stories" rather than a rigorous science.

This paper introduces the Explanatory Virtues Framework, a rubric derived from the philosophy of science to evaluate, compare, and improve AI explanations by looking at traits like simplicity, unification, and "hard-to-varyness."

The Explanatory Virtues Framework

The authors synthesize four major philosophical perspectives into a unified framework for MI. This framework distinguishes between empirical values (how well an explanation fits the data) and theoretical values (the internal properties of the explanation itself).

A map of the relationships between empirical and theoretical explanatory values, highlighting key virtues like Accuracy, Precision, and Unification.

The framework categorizes virtues into four pillars:

  1. Bayesian Virtues: Grounded in information theory, these focus on metrics like Co-explanation (explaining multiple data points simultaneously) and Unification.
  2. Kuhnian Virtues: These represent classic scientific values such as Simplicity, Consistency, and Fruitfulness.
  3. Deutschian Virtues: This introduces the concept of Hard-to-Varyness—the principle that a good explanation should be difficult to "tweak" to fit new data without breaking its core logic.
  4. Nomological Virtues: These evaluate whether an explanation appeals to "general laws" or universal principles of neural networks, moving beyond isolated feature catalogs.
Formal definitions for Kuhnian virtues including Fruitfulness, Consistency, Parsimony, Conciseness, and Complexity.

Key Insights: From Description to Proof

A central argument of the paper is the "Mechanistic Turn": MI is evolving from a descriptive stage (cataloging neurons) to a causal stage (proving mechanisms).

  • Compact Proofs as the Gold Standard: The framework identifies "Compact Proofs"—where an explanation is converted into a formal mathematical guarantee of model performance—as the most virtuous method. These proofs naturally enforce a trade-off between accuracy and simplicity.
  • Simplicity vs. Parsimony: The authors clarify that "Parsimony" (fewer features) is often less valuable than "Conciseness" (shorter information-theoretic description length).
  • Nomothetic Approach: Instead of "stamp collecting" (naming every feature), the authors advocate for finding universal principles that apply across different architectures.

Evaluating Current MI Methods

The paper provides a qualitative "report card" for existing techniques:

  • Clustering: Scores poorly due to a lack of causal depth and failure to unify behaviors.
  • Sparse Autoencoders (SAEs): Strong on simplicity and accuracy, but often produce "ad-hoc" explanations that are too easy to vary.
  • Causal Circuits: Excellent at demonstrating causality but often fail at Unification; a circuit for one task rarely explains another.
  • Compact Proofs: Score the highest across most virtues, providing falsifiable, hard-to-vary, and precise explanations.

Limitations and Future Directions

Despite the framework's strengths, several challenges remain:

  • Scaling: Compact Proofs are difficult to scale to massive Large Language Models (LLMs) due to "superposition noise."
  • The Human-AI Gap: A mathematically perfect proof might be incomprehensible to a human. The authors propose Local Decodability—the ability to understand parts of an explanation in isolation—as a future virtue.
  • Defining Simplicity: A universally accepted mathematical definition for "simplicity" in neural networks remains an open research question.

Summary updated

Abstract

Mechanistic Interpretability (MI) aims to understand neural networks through causal explanations. Though MI has many explanation-generating methods, progress has been limited by the lack of a universal approach to evaluating explanations. Here we analyse the fundamental question "What makes a good explanation?" We introduce a pluralist Explanatory Virtues Framework drawing on four perspectives from the Philosophy of Science - the Bayesian, Kuhnian, Deutschian, and Nomological - to systematically evaluate and improve explanations in MI. We find that Compact Proofs consider many explanatory virtues and are hence a promising approach. Fruitful research directions implied by our framework include (1) clearly defining explanatory simplicity, (2) focusing on unifying explanations and (3) deriving universal principles for neural networks. Improved MI methods enhance our ability to monitor, predict, and steer AI systems.
BibTeX
@misc{ayonrinde2025evaluating,
  author = {Kola Ayonrinde and Louis Jaburi},
  title = {{Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability - The Strange Science Part I.ii}},
  year = {2025},
  eprint = {2505.01372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.01372},
}
Figure