TL;DR
Can Large Language Models (LLMs) explain the predictions of other machine learning models? This paper introduces the first systematic framework evaluating LLMs as post hoc explainers for tabular classifiers via in-context learning (ICL). The authors show that prompting LLMs with local perturbations or a handful of example explanations allows them to match or exceed the faithfulness of classical explainers like LIME and SHAP—identifying the most critical feature with over 72% accuracy.
What Problem Does This Paper Address?
Explaining the decisions of complex black-box predictive models is crucial in high-stakes fields like healthcare, criminal justice, and finance. While feature attribution methods such as LIME, SHAP, and Integrated Gradients are widely used, they carry distinct practical drawbacks:
- Many require white-box gradient access, which is unavailable for closed-source systems.
- Perturbation-based approximations can be computationally expensive or hyperparameter-sensitive.
- Different explainers often produce conflicting explanations for the same prediction.
Given the strong reasoning and pattern-matching abilities of modern LLMs, the authors ask: Can an LLM act as an off-the-shelf post hoc explainer without needing model weights, training, or fine-tuning?
Methods
The authors propose a framework exploring four prompting strategies that feed varying levels of local neighborhood information or prior explanations into an LLM:
- Perturbation-based ICL: The LLM receives the test sample and $n_{\text{ICL}}$ randomly or high-confidence sampled perturbations from its local neighborhood alongside the black-box model’s predicted classes. It is asked to rank the top-$k$ most important features.
- Prediction-based ICL: Similar to standard ICL classification, the LLM first predicts the label of the unlabelled test sample based on neighboring perturbations, and then explains which features drove that prediction.
- Instruction-based ICL: Instead of raw inputs, perturbations are formatted as relative differences ($\Delta \text{Input}, \Delta \text{Output}$) w.r.t. the test sample. The prompt guides the model with step-by-step instructions: analyze positive versus negative correlation per feature, score feature importance from 0–100, and update a running rank.
- Explanation-based ICL: The LLM is provided with just four demonstration examples ($n_{\text{ICL}} = 4$) containing inputs, predicted labels, and ground-truth explanations from an existing explainer (e.g., LIME or Integrated Gradients), querying the LLM to generate attributions for an unseen test point.
Faithfulness is measured using Feature Agreement (FA) and Rank Agreement (RA) against true model coefficients for Logistic Regression, as well as Prediction Gap on Important (PGI) and Unimportant (PGU) feature perturbations for Artificial Neural Networks (ANNs).
Datasets
The authors evaluate models on four standard tabular classification benchmarks:
- Blood Transfusion Service Center (4 features)
- COMPAS Recidivism (6 features)
- UCI Default Credit (10 features)
- Adult Income (13 features)
Evaluated predictive models include Logistic Regression (LR) and multi-layer Artificial Neural Networks (ANN). Evaluated LLMs are GPT-3.5 and GPT-4.
Key Insights and Ideas
- Prompting format matters more than raw data volume: Supplying explicit feature deltas (relative changes w.r.t. the base sample) drastically improves faithfulness over raw absolute feature values. It relieves the LLM from having to calculate differences internally.
- More context hurts (the "Lost in the Middle" trap): While traditional explainers like LIME improve when sampling thousands of perturbations, LLM performance peaks around 12–16 samples and degrades at 64 samples due to context-length distraction.
- Rapid emulation via meta-explanations: With only 4 few-shot demonstrations, LLMs successfully emulate existing post hoc explainers (Grad, LIME, Integrated Gradients) on test instances.
- LLMs inject helpful domain priors: When mimicking weaker baselines like Gradient $\times$ Input (ITG) or SHAP on linear models, LLM-generated explanations actually outperformed the base explainers themselves, suggesting LLM world knowledge helped correct noisy attributions.
Key Results
- Top-1 Feature Identification: Across all four datasets and two model families, LLMs correctly identify the single most influential feature (top-$k=1$) with 72.19% Feature Agreement, vastly outperforming SHAP (29.75%), ITG (29.50%), and random baselines (15.00%).
- ANN Faithfulness: On complex neural networks, LLM prompting strategies perform on par with white-box gradient methods and LIME. On larger datasets (Adult and Credit), LLMs achieve 51.74% lower PGU (perturbing unimportant features causes less damage) and 163.40% higher PGI than SHAP, ITG, and random baselines.
- GPT-4 vs. GPT-3.5: GPT-4 consistently generates more faithful attributions, yielding 4.53% higher FA and 48.01% higher RA over GPT-3.5 on the Adult dataset.
Limitations and Future Work
- Inference Latency & Cost: Generating explanations for 100 samples with an LLM took roughly 1,550 seconds, compared to 0.02–10 seconds for traditional algorithms like LIME or Vanilla Gradients.
- Performance Drop on Full Rankings: While LLMs excel at finding the top-1 feature, their rank agreement degrades substantially for $k > 1$, particularly when features have nearly identical importance or weights.
- Instruction Brittleness: Although rare, the models occasionally violated output formatting constraints or outputted "insufficient information" disclaimers, especially at higher sampling temperatures.
- Future Work: Developing dedicated open-source, fine-tuned explainer LLMs to cut API inference latency, and scaling the method beyond tabular inputs to image and multimodal domains.
Summary updated