ALMANACS: A Simulatability Benchmark for Language Model Explainability

Edmund Mills, Shiye Su, Stuart Russell, Scott EmmonsSubmission to ICLR 2025 · Rejected

TL;DR

ALMANACS is a fully automated benchmark designed to evaluate language model explainability through simulatability—measuring whether an explanation helps an observer predict how a model will respond to novel, held-out inputs. Across twelve safety-relevant topics and four standard interpretability techniques, the findings are sobering: no explanation method consistently outperforms a simple, explanation-free baseline.

What Problem Does This Paper Address?

Despite an explosion of interpretability and explainability methods for deep neural networks, the field lacks standardized, apples-to-apples evaluation. Most interpretability methods are validated on bespoke tasks or depend on costly, slow human user studies that cannot scale to modern large language models (LLMs). Furthermore, conceptual desiderata like "faithfulness" (whether an explanation accurately reflects internal reasoning) and "completeness" are notoriously difficult to quantify.

The authors address this gap by focusing on simulatability: if an explanation truly reflects a model's internal decision-making process, an observer should be able to use it to accurately predict that model’s behavior on unseen, distributionally shifted inputs.

Methods

ALMANACS introduces an automated Explainer / Predictor framework:

  1. The Explainer ($\mathcal{E}$): Takes target model $f$ and dataset $\mathcal{D} = \{(x, y)\}$ (where $y \in [0, 1]$ is the normalized probability of answering "Yes") and attaches an explanation $e$ to each training instance, producing an augmented dataset $\tilde{\mathcal{D}} = \{(x, y, e)\}$.
  2. The Predictor ($\mathcal{P}$): An independent observer that reads 10 retrieved nearest-neighbor examples from $\tilde{\mathcal{D}}$ (via Sentence-BERT embeddings) to predict the model's output $\tilde{y}$ on a held-out test input $x$. Crucially, $\mathcal{P}$ never sees explanations for the test question, preventing label leakage.
  3. Automated LLM Predictor: ALMANACS uses GPT-4 as the predictor, enabling fast, zero-marginal-cost evaluations.
  4. Evaluated Methods:
  5. Naive baselines: PredictAverage, NearestNeighbor, NearestNeighbor3, and LogisticRegression.
  6. Control: NoExpl (10-shot in-context learning of input-output pairs without explanations).
  7. Interpretability methods: Counterfactual (nearest neighbor with $|\Delta y| > 0.2$), Rationalization (model-generated step-by-step reasoning), Attention (verbalized top-25 salient tokens from the final layer), and Integrated Gradients (verbalized top-25 attribution tokens).
  8. Metrics: Kullback–Leibler divergence (KLDIV, a proper scoring rule), Total Variation Distance (TVDIST), and Spearman’s $\rho$.

Datasets

The authors construct ALMANACS (Anticipating Language Model Answers in Non-objective And Complex Scenarios):

  • 12 Safety-Relevant Topics: Covering ethical dilemmas, advanced AI risks, power-seeking, sycophancy, harmful requests, situational awareness, and misinformation.
  • Templated Structure & Distributional Shift: Each topic contains 15 parameterized templates with 5 placeholder slots (15 possible values each). Specific combinations and values are strictly held out for the test set to ensure models must extrapolate rather than interpolate.
  • Adversarial Filtering: Templates are filtered against a logistic regression baseline to ensure complex, nonlinear behavior.
  • Subject Models: The benchmark evaluates flan-alpaca-gpt4-xl (3B encoder-decoder) and vicuna-7b-v1.3 (7B decoder-only), comprising 180,000 train and 18,000 test examples per model.

Key Insights and Ideas

  • Explanations Fail to Aid Simulatability: Averaged across all topics, neither feature attributions (Attention, Integrated Gradients), counterfactual examples, nor self-rationalizations reliably beat the NoExpl control.
  • Validating the LLM-as-Predictor: When evaluated on a synthetic linear model with known ground-truth weights, providing actual weight explanations dropped GPT-4's prediction error (KLDIV) from 0.54 down to 0.16. This confirms that GPT-4 can exploit helpful explanations; the current explainability methods simply fail to provide actionable signal.
  • Agreement with Human Evaluators: In a 400-hour study across 10 human annotators, humans likewise showed no statistically significant benefit from explanations over the NoExpl control in aggregate, validating GPT-4 as an effective simulation proxy.
  • LLMs Out-Simulate Humans: GPT-4 was generally more accurate at predicting model behavior than human evaluators, likely due to superior few-shot in-context pattern matching.

Key Results

Predictor Method flan-alpaca-gpt4-xl (Mean KLDIV ↓) vicuna-7b-v1.3 (Mean KLDIV ↓)
PredictAverage 0.15 0.17
LogisticRegression 0.11 0.09
NoExpl (Control) 0.10 0.08
Counterfactual 0.09 0.08
Rationalization 0.10 0.10
Attention 0.10 0.08
Integrated Gradients 0.10 0.09
  • On flan-alpaca-gpt4-xl, explanation methods tied the control at 0.09–0.10 mean KLDIV.
  • On vicuna-7b-v1.3, explanation methods either matched (0.08) or performed worse than (0.09–0.10) the control.
  • Qualitative inspections revealed that salience methods often highlight superficial question formatting words (e.g., "would", "choose"), while rationalizations focus narrowly on scenario specifics rather than generalizable decision rules.

Limitations and Future Work

  • Not a Complete Substitute for Humans: While human and LLM trends match in the aggregate, humans benefited from rationalizations on isolated subtasks (e.g., Hiring Decisions) where GPT-4 did not, showing that automated evaluation may produce false negatives.
  • Binary Scope: ALMANACS is restricted to binary probability predictions on structured Yes/No questions, leaving open-ended text generation unaddressed.
  • Lossy Verbalizations: Heuristically formatting continuous attributions (e.g., top-25 tokens) or raw weights into text may restrict an explanation's utility.
  • Open Challenge: Developing scalable, faithful explainability methods that genuinely help external observers predict model behavior under distributional shift remains an unsolved problem.

Summary updated

Abstract

How do we measure the efficacy of language model explainability methods? While many explainability methods have been developed, they are typically evaluated on bespoke tasks, preventing an apples-to-apples comparison. To help fill this gap, we present ALMANACS, a language model explainability benchmark. ALMANACS scores explainability methods on simulatability, i.e., how well the explanations improve behavior prediction on new inputs. The ALMANACS scenarios span twelve safety-relevant topics such as ethical reasoning and advanced AI behaviors; they have idiosyncratic premises to invoke model-specific behavior; and they have a train-test distributional shift to encourage faithful explanations. By using another language model to predict behavior based on the explanations, ALMANACS is a fully automated benchmark. While not a replacement for human evaluations, we aim for ALMANACS to be a complementary, automated tool that allows for fast, scalable evaluation. Using ALMANACS, we evaluate counterfactual, rationalization, attention, and Integrated Gradients explanations. Our results are sobering: when averaged across all topics, no explanation method outperforms the explanation-free control. We conclude that despite modest successes in prior work, developing an explanation method that aids simulatability in ALMANACS remains an open challenge.
BibTeX
@misc{mills2023almanacs,
  author = {Edmund Mills and Shiye Su and Stuart Russell and Scott Emmons},
  title = {{ALMANACS: A Simulatability Benchmark for Language Model Explainability}},
  year = {2023},
  eprint = {2312.12747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.12747},
}

Loading related content...

Figure