Are Large Language Models Post Hoc Explainers?

Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, Himabindu LakkarajuSubmission to ICLR 2024 · Rejected

TL;DR

Can Large Language Models (LLMs) explain the predictions of other machine learning models? This paper introduces the first systematic framework evaluating LLMs as post hoc explainers for tabular classifiers via in-context learning (ICL). The authors show that prompting LLMs with local perturbations or a handful of example explanations allows them to match or exceed the faithfulness of classical explainers like LIME and SHAP—identifying the most critical feature with over 72% accuracy.

What Problem Does This Paper Address?

Explaining the decisions of complex black-box predictive models is crucial in high-stakes fields like healthcare, criminal justice, and finance. While feature attribution methods such as LIME, SHAP, and Integrated Gradients are widely used, they carry distinct practical drawbacks:

  • Many require white-box gradient access, which is unavailable for closed-source systems.
  • Perturbation-based approximations can be computationally expensive or hyperparameter-sensitive.
  • Different explainers often produce conflicting explanations for the same prediction.

Given the strong reasoning and pattern-matching abilities of modern LLMs, the authors ask: Can an LLM act as an off-the-shelf post hoc explainer without needing model weights, training, or fine-tuning?

Methods

The authors propose a framework exploring four prompting strategies that feed varying levels of local neighborhood information or prior explanations into an LLM:

  1. Perturbation-based ICL: The LLM receives the test sample and $n_{\text{ICL}}$ randomly or high-confidence sampled perturbations from its local neighborhood alongside the black-box model’s predicted classes. It is asked to rank the top-$k$ most important features.
  2. Prediction-based ICL: Similar to standard ICL classification, the LLM first predicts the label of the unlabelled test sample based on neighboring perturbations, and then explains which features drove that prediction.
  3. Instruction-based ICL: Instead of raw inputs, perturbations are formatted as relative differences ($\Delta \text{Input}, \Delta \text{Output}$) w.r.t. the test sample. The prompt guides the model with step-by-step instructions: analyze positive versus negative correlation per feature, score feature importance from 0–100, and update a running rank.
  4. Explanation-based ICL: The LLM is provided with just four demonstration examples ($n_{\text{ICL}} = 4$) containing inputs, predicted labels, and ground-truth explanations from an existing explainer (e.g., LIME or Integrated Gradients), querying the LLM to generate attributions for an unseen test point.

Faithfulness is measured using Feature Agreement (FA) and Rank Agreement (RA) against true model coefficients for Logistic Regression, as well as Prediction Gap on Important (PGI) and Unimportant (PGU) feature perturbations for Artificial Neural Networks (ANNs).

Datasets

The authors evaluate models on four standard tabular classification benchmarks:

  • Blood Transfusion Service Center (4 features)
  • COMPAS Recidivism (6 features)
  • UCI Default Credit (10 features)
  • Adult Income (13 features)

Evaluated predictive models include Logistic Regression (LR) and multi-layer Artificial Neural Networks (ANN). Evaluated LLMs are GPT-3.5 and GPT-4.

Key Insights and Ideas

  • Prompting format matters more than raw data volume: Supplying explicit feature deltas (relative changes w.r.t. the base sample) drastically improves faithfulness over raw absolute feature values. It relieves the LLM from having to calculate differences internally.
  • More context hurts (the "Lost in the Middle" trap): While traditional explainers like LIME improve when sampling thousands of perturbations, LLM performance peaks around 12–16 samples and degrades at 64 samples due to context-length distraction.
  • Rapid emulation via meta-explanations: With only 4 few-shot demonstrations, LLMs successfully emulate existing post hoc explainers (Grad, LIME, Integrated Gradients) on test instances.
  • LLMs inject helpful domain priors: When mimicking weaker baselines like Gradient $\times$ Input (ITG) or SHAP on linear models, LLM-generated explanations actually outperformed the base explainers themselves, suggesting LLM world knowledge helped correct noisy attributions.

Key Results

  • Top-1 Feature Identification: Across all four datasets and two model families, LLMs correctly identify the single most influential feature (top-$k=1$) with 72.19% Feature Agreement, vastly outperforming SHAP (29.75%), ITG (29.50%), and random baselines (15.00%).
  • ANN Faithfulness: On complex neural networks, LLM prompting strategies perform on par with white-box gradient methods and LIME. On larger datasets (Adult and Credit), LLMs achieve 51.74% lower PGU (perturbing unimportant features causes less damage) and 163.40% higher PGI than SHAP, ITG, and random baselines.
  • GPT-4 vs. GPT-3.5: GPT-4 consistently generates more faithful attributions, yielding 4.53% higher FA and 48.01% higher RA over GPT-3.5 on the Adult dataset.

Limitations and Future Work

  • Inference Latency & Cost: Generating explanations for 100 samples with an LLM took roughly 1,550 seconds, compared to 0.02–10 seconds for traditional algorithms like LIME or Vanilla Gradients.
  • Performance Drop on Full Rankings: While LLMs excel at finding the top-1 feature, their rank agreement degrades substantially for $k > 1$, particularly when features have nearly identical importance or weights.
  • Instruction Brittleness: Although rare, the models occasionally violated output formatting constraints or outputted "insufficient information" disclaimers, especially at higher sampling temperatures.
  • Future Work: Developing dedicated open-source, fine-tuned explainer LLMs to cut API inference latency, and scaling the method beyond tabular inputs to image and multimodal domains.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly used as powerful tools for a plethora of natural language processing (NLP) applications. A recent innovation, in-context learning (ICL), enables LLMs to learn new tasks by supplying a few examples in the prompt during inference time, thereby eliminating the need for model fine-tuning. While LLMs have been utilized in several applications, their applicability in explaining the behavior of other models remains relatively unexplored. Despite the growing number of new explanation techniques, many require white-box access to the model and/or are computationally expensive, highlighting a need for next-generation post hoc explainers. In this work, we present the first framework to study the effectiveness of LLMs in explaining other predictive models. More specifically, we propose a novel framework encompassing multiple prompting strategies: i) Perturbation-based ICL, ii) Prediction-based ICL, iii) Instruction-based ICL, and iv) Explanation-based ICL, with varying levels of information about the underlying ML model and the local neighborhood of the test sample. We conduct extensive experiments with real-world benchmark datasets to demonstrate that LLM generated explanations perform on par with state-of-the-art post hoc explainers using their ability to leverage ICL examples and their internal knowledge in generating model explanations. On average, across four datasets and two ML models, we observe that LLMs identify the most important feature with 72.19% accuracy, opening up new frontiers in explainable artificial intelligence (XAI) to explore LLM-based explanation frameworks.
BibTeX
@inproceedings{kroeger2023are,
  author = {Nicholas Kroeger and Dan Ley and Satyapriya Krishna and Chirag Agarwal and Himabindu Lakkaraju},
  title = {{Are Large Language Models Post Hoc Explainers?}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=1G7n7LW3mF},
}

Loading related content...

Figure