Are Large Language Models Post Hoc Explainers?

Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, Himabindu Lakkaraju

Abstract

Recent advancements in Large Language Models (LLMs) have demonstrated exceptional capabilities in complex tasks like machine translation and commonsense reasoning. One of the primary reasons for their adaptability is In-Context Learning (ICL), which allows LLMs to perform new tasks using only a few samples in a prompt. This work explores the effectiveness of LLMs in explaining other complex predictive models using ICL. We propose the In-Context Explainers (ICE) framework, comprising three novel approaches that leverage ICL to generate post hoc explanations for tabular and text models. Our analysis on real-world datasets demonstrates that LLMs can explain predictive models with faithfulness comparable to state-of-the-art explainers like LIME and SHAP.

Introduction

As machine learning models become increasingly complex "black boxes," the need for post hoc explanations—tools that clarify why a model made a specific prediction—has become critical. Traditional methods like LIME or SHAP are effective but computationally expensive, often requiring thousands of queries to approximate local behavior. This paper investigates whether LLMs can act as independent explainers by observing a small set of input-output pairs from a target model to deduce feature importance.

The In-Context Explainers (ICE) Framework

The ICE framework utilizes three distinct prompting strategies to elicit explanations from LLMs:

  1. Perturb ICL (P-ICL): The LLM is provided with a model description, a small dataset of 16 local perturbations (showing how input changes affect output), and a question asking for the top-K features.
  2. Perturb+Guide ICL (PG-ICL): This approach adds a structured "thinking framework" to P-ICL. The prompt instructs the LLM to analyze features individually, rate their importance (0-100), and maintain a running rank, effectively implementing a Chain-of-Thought process.
  3. Explain ICL (E-ICL): A mimicry strategy where the LLM is shown a few examples of (Input, Output, Explanation) triplets, where the explanation is sourced from a gold-standard method like LIME. The LLM then learns to generate similar explanations for new samples.
The ICE framework workflow: from dataset and model to prompt generation, LLM call, and final explanation metrics.

To handle tabular data, the framework uses serialization templates to convert numerical values into natural language strings suitable for LLM processing.

Experimental Setup

The framework was evaluated across eight datasets, including tabular data (e.g., Blood Transfusion, Credit Default, HELOC) and text data (Amazon, IMDb, and Yelp reviews). The target "black box" models included Logistic Regression (LR), multi-layer Artificial Neural Networks (ANNs), and Transformers. The performance was measured using metrics like Feature Agreement (FA), Rank Agreement (RA), and Prototypicality (PGU/PGI).

Key Results and Insights

Performance vs. Traditional Explainers

LLM-based explanations proved to be remarkably faithful. In many scenarios, ICE methods matched or exceeded the performance of specialized tools like LIME. For larger tabular datasets, LLM explanations achieved significantly better results than standard gradient-based baselines, showing a 50.13% lower PGU (better at ignoring noise) and 144.82% higher PGI (better at identifying key features).

Performance comparison across different models and datasets, showing ICE methods (P-ICL, PG-ICL, E-ICL) competing with traditional explainers like LIME and SHAP.

The Relational Advantage

A critical finding was that LLMs perform better when shown "raw perturbations"—the specific change in input and the resulting change in output—rather than absolute values. This relational view allows the model to focus on the delta directly, bypassing complex mental arithmetic.

Model Scaling and Guidance

There is a clear intelligence threshold for these tasks; GPT-4 significantly outperformed GPT-3.5. Furthermore, the structured analytical process in PG-ICL led to more consistent and higher faithfulness scores than generic instructions, particularly on complex tabular data.

Limitations

Despite their success, LLM explainers face challenges:

  • Alphabetical Bias: When features have similar importance, LLMs sometimes default to alphabetical ranking.
  • Prompt Sensitivity: Results are highly dependent on the specific wording and serialization templates used.
  • Cost: High-end LLM API calls can be more expensive and slower than local execution of traditional algorithms.

Conclusion

This research demonstrates that LLMs are capable of acting as effective post hoc explainers for other predictive models. By leveraging ICL and structured prompting, the ICE framework provides a promising new avenue for model interpretability, potentially reducing the computational overhead of traditional perturbation-based methods while maintaining high explanation faithfulness.

Summary updated

Abstract

Recent advancements in Large Language Models (LLMs) have demonstrated exceptional capabilities in complex tasks like machine translation, commonsense reasoning, and language understanding. One of the primary reasons for the adaptability of LLMs in such diverse tasks is their in-context learning (ICL) capability, which allows them to perform well on new tasks by simply using a few task samples in the prompt. Despite their effectiveness in enhancing the performance of LLMs on diverse language and tabular tasks, these methods have not been thoroughly explored for their potential to generate post hoc explanations. In this work, we carry out one of the first explorations to analyze the effectiveness of LLMs in explaining other complex predictive models using ICL. To this end, we propose a novel framework, In-Context Explainers, comprising of three novel approaches that exploit the ICL capabilities of LLMs to explain the predictions made by other predictive models. We conduct extensive analysis with these approaches on real-world tabular and text datasets and demonstrate that LLMs are capable of explaining other predictive models similar to state-of-the-art post hoc explainers, opening up promising avenues for future research into LLM-based post hoc explanations of complex predictive models.
BibTeX
@inproceedings{kroeger2023are,
  author = {Nicholas Kroeger and Dan Ley and Satyapriya Krishna and Chirag Agarwal and Himabindu Lakkaraju},
  title = {{Are Large Language Models Post Hoc Explainers?}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=1G7n7LW3mF},
}
Figure