Abstract
How do we measure the efficacy of language model explainability methods? While many explainability methods have been developed, they are typically evaluated on bespoke tasks, preventing an apples-to-apples comparison. To help fill this gap, we present ALMANACS, a language model explainability benchmark. ALMANACS scores explainability methods on simulatability, i.e., how well the explanations improve behavior prediction on new inputs. The ALMANACS scenarios span twelve safety-relevant topics such as ethical reasoning and advanced AI behaviors; they have idiosyncratic premises to invoke model-specific behavior; and they have a train-test distributional shift to encourage faithful explanations. By using another language model to predict behavior based on the explanations, ALMANACS is a fully automated benchmark. While not a replacement for human evaluations, we aim for ALMANACS to be a complementary, automated tool that allows for fast, scalable evaluation. Using ALMANACS, we evaluate counterfactual, rationalization, attention, and Integrated Gradients explanations. Our results are sobering: when averaged across all topics, no explanation method outperforms the explanation-free control. We conclude that despite modest successes in prior work, developing an explanation method that aids simulatability in ALMANACS remains an open challenge.
Introduction
As large language models (LLMs) are deployed in increasingly sensitive areas, understanding the internal logic behind their decisions is critical for safety. However, the field of interpretability currently lacks standardized evaluation metrics. Most researchers evaluate their explainability methods on bespoke tasks, making cross-method comparisons difficult. Furthermore, the "gold standard" of human evaluation is slow and unscalable. ALMANACS addresses this by providing a fully automated benchmark based on "simulatability."
The ALMANACS Framework
ALMANACS measures the efficacy of an explanation by its ability to help a Predictor model anticipate a Target Model's behavior on new, unseen inputs. If an explanation is truly faithful and complete, the Predictor should accurately guess the target model's response probability.
The pipeline involves:
- The Explainer: Generates explanations such as counterfactuals (similar inputs with different outputs), rationalizations (model-generated justifications), or saliency maps (Attention and Integrated Gradients).
- The Predictor: An automated model (e.g., GPT-4) that receives few-shot examples of (Question, Answer, Explanation) and predicts the target model's probability of answering "Yes" to a test question.
To ensure the Predictor doesn't simply memorize the target model, ALMANACS introduces a distributional shift between training and test sets, forcing the Predictor to rely on the provided explanations.
Benchmark Datasets
The benchmark covers 12 safety-relevant topics, including ethical reasoning (Moral Dilemmas), sycophancy, power-seeking, and harmful requests. Each topic contains complex scenarios designed to elicit idiosyncratic model behaviors, ensuring the Predictor cannot rely solely on "common sense."
The dataset includes 180,000 training and 18,000 test examples across two target models: flan-alpaca-gpt4-xl (3B) and vicuna-7b-v1.3 (7B).
Key Results: A Reality Check for Explanations
The primary metric for simulatability is the Kullback–Leibler divergence (KLDIV) between the predicted and actual response probabilities. A lower KLDIV indicates better simulation.
- The Baseline is Hard to Beat: Surprisingly, the "No-Explanation" (NOEXPL) control performed remarkably well. For
vicuna-7b, the baseline KLDIV was 0.08. - Explanations Fail to Add Value: None of the tested methods—Counterfactuals, Rationalizations, or Saliency maps—consistently outperformed the NOEXPL baseline. Most hovered between 0.08 and 0.10. In some instances, such as "Situational Awareness" for Vicuna, adding rationalizations significantly degraded performance (KLDIV increased to 0.27).
- Saliency Map Limitations: Qualitative analysis revealed that saliency methods often highlight structural tokens (e.g., "would", "choose") rather than the semantic nuances driving the model's decision.
- LMs as Superior Evaluators: GPT-4 outperformed human annotators in predicting model behavior, suggesting that LMs are better at picking up on the idiosyncratic biases of other models through in-context learning.
Conclusion
ALMANACS provides a rigorous, automated reality check for LM explainability. The finding that current popular explanation methods fail to improve simulatability over a simple baseline highlights a significant gap in the field. While ALMANACS is a complement to human study and currently focuses on binary classification, it sets a high bar for future interpretability research. Developing explanation methods that are truly informative and aid in model simulation remains an open and urgent challenge.
Summary updated