TL;DR
ALMANACS is a fully automated benchmark designed to evaluate language model explainability through simulatability—measuring whether an explanation helps an observer predict how a model will respond to novel, held-out inputs. Across twelve safety-relevant topics and four standard interpretability techniques, the findings are sobering: no explanation method consistently outperforms a simple, explanation-free baseline.
What Problem Does This Paper Address?
Despite an explosion of interpretability and explainability methods for deep neural networks, the field lacks standardized, apples-to-apples evaluation. Most interpretability methods are validated on bespoke tasks or depend on costly, slow human user studies that cannot scale to modern large language models (LLMs). Furthermore, conceptual desiderata like "faithfulness" (whether an explanation accurately reflects internal reasoning) and "completeness" are notoriously difficult to quantify.
The authors address this gap by focusing on simulatability: if an explanation truly reflects a model's internal decision-making process, an observer should be able to use it to accurately predict that model’s behavior on unseen, distributionally shifted inputs.
Methods
ALMANACS introduces an automated Explainer / Predictor framework:
- The Explainer ($\mathcal{E}$): Takes target model $f$ and dataset $\mathcal{D} = \{(x, y)\}$ (where $y \in [0, 1]$ is the normalized probability of answering "Yes") and attaches an explanation $e$ to each training instance, producing an augmented dataset $\tilde{\mathcal{D}} = \{(x, y, e)\}$.
- The Predictor ($\mathcal{P}$): An independent observer that reads 10 retrieved nearest-neighbor examples from $\tilde{\mathcal{D}}$ (via Sentence-BERT embeddings) to predict the model's output $\tilde{y}$ on a held-out test input $x$. Crucially, $\mathcal{P}$ never sees explanations for the test question, preventing label leakage.
- Automated LLM Predictor: ALMANACS uses GPT-4 as the predictor, enabling fast, zero-marginal-cost evaluations.
- Evaluated Methods:
- Naive baselines:
PredictAverage,NearestNeighbor,NearestNeighbor3, andLogisticRegression. - Control:
NoExpl(10-shot in-context learning of input-output pairs without explanations). - Interpretability methods:
Counterfactual(nearest neighbor with $|\Delta y| > 0.2$),Rationalization(model-generated step-by-step reasoning),Attention(verbalized top-25 salient tokens from the final layer), andIntegrated Gradients(verbalized top-25 attribution tokens). - Metrics: Kullback–Leibler divergence (KLDIV, a proper scoring rule), Total Variation Distance (TVDIST), and Spearman’s $\rho$.
Datasets
The authors construct ALMANACS (Anticipating Language Model Answers in Non-objective And Complex Scenarios):
- 12 Safety-Relevant Topics: Covering ethical dilemmas, advanced AI risks, power-seeking, sycophancy, harmful requests, situational awareness, and misinformation.
- Templated Structure & Distributional Shift: Each topic contains 15 parameterized templates with 5 placeholder slots (15 possible values each). Specific combinations and values are strictly held out for the test set to ensure models must extrapolate rather than interpolate.
- Adversarial Filtering: Templates are filtered against a logistic regression baseline to ensure complex, nonlinear behavior.
- Subject Models: The benchmark evaluates
flan-alpaca-gpt4-xl(3B encoder-decoder) andvicuna-7b-v1.3(7B decoder-only), comprising 180,000 train and 18,000 test examples per model.
Key Insights and Ideas
- Explanations Fail to Aid Simulatability: Averaged across all topics, neither feature attributions (Attention, Integrated Gradients), counterfactual examples, nor self-rationalizations reliably beat the
NoExplcontrol. - Validating the LLM-as-Predictor: When evaluated on a synthetic linear model with known ground-truth weights, providing actual weight explanations dropped GPT-4's prediction error (KLDIV) from 0.54 down to 0.16. This confirms that GPT-4 can exploit helpful explanations; the current explainability methods simply fail to provide actionable signal.
- Agreement with Human Evaluators: In a 400-hour study across 10 human annotators, humans likewise showed no statistically significant benefit from explanations over the
NoExplcontrol in aggregate, validating GPT-4 as an effective simulation proxy. - LLMs Out-Simulate Humans: GPT-4 was generally more accurate at predicting model behavior than human evaluators, likely due to superior few-shot in-context pattern matching.
Key Results
| Predictor Method | flan-alpaca-gpt4-xl (Mean KLDIV ↓) |
vicuna-7b-v1.3 (Mean KLDIV ↓) |
|---|---|---|
| PredictAverage | 0.15 | 0.17 |
| LogisticRegression | 0.11 | 0.09 |
| NoExpl (Control) | 0.10 | 0.08 |
| Counterfactual | 0.09 | 0.08 |
| Rationalization | 0.10 | 0.10 |
| Attention | 0.10 | 0.08 |
| Integrated Gradients | 0.10 | 0.09 |
- On
flan-alpaca-gpt4-xl, explanation methods tied the control at 0.09–0.10 mean KLDIV. - On
vicuna-7b-v1.3, explanation methods either matched (0.08) or performed worse than (0.09–0.10) the control. - Qualitative inspections revealed that salience methods often highlight superficial question formatting words (e.g., "would", "choose"), while rationalizations focus narrowly on scenario specifics rather than generalizable decision rules.
Limitations and Future Work
- Not a Complete Substitute for Humans: While human and LLM trends match in the aggregate, humans benefited from rationalizations on isolated subtasks (e.g., Hiring Decisions) where GPT-4 did not, showing that automated evaluation may produce false negatives.
- Binary Scope: ALMANACS is restricted to binary probability predictions on structured Yes/No questions, leaving open-ended text generation unaddressed.
- Lossy Verbalizations: Heuristically formatting continuous attributions (e.g., top-25 tokens) or raw weights into text may restrict an explanation's utility.
- Open Challenge: Developing scalable, faithful explainability methods that genuinely help external observers predict model behavior under distributional shift remains an unsolved problem.
Summary updated