Abstract
Recent advancements in Large Language Models (LLMs) have demonstrated exceptional capabilities in complex tasks like machine translation and commonsense reasoning. One of the primary reasons for their adaptability is In-Context Learning (ICL), which allows LLMs to perform new tasks using only a few samples in a prompt. This work explores the effectiveness of LLMs in explaining other complex predictive models using ICL. We propose the In-Context Explainers (ICE) framework, comprising three novel approaches that leverage ICL to generate post hoc explanations for tabular and text models. Our analysis on real-world datasets demonstrates that LLMs can explain predictive models with faithfulness comparable to state-of-the-art explainers like LIME and SHAP.
Introduction
As machine learning models become increasingly complex "black boxes," the need for post hoc explanations—tools that clarify why a model made a specific prediction—has become critical. Traditional methods like LIME or SHAP are effective but computationally expensive, often requiring thousands of queries to approximate local behavior. This paper investigates whether LLMs can act as independent explainers by observing a small set of input-output pairs from a target model to deduce feature importance.
The In-Context Explainers (ICE) Framework
The ICE framework utilizes three distinct prompting strategies to elicit explanations from LLMs:
- Perturb ICL (P-ICL): The LLM is provided with a model description, a small dataset of 16 local perturbations (showing how input changes affect output), and a question asking for the top-K features.
- Perturb+Guide ICL (PG-ICL): This approach adds a structured "thinking framework" to P-ICL. The prompt instructs the LLM to analyze features individually, rate their importance (0-100), and maintain a running rank, effectively implementing a Chain-of-Thought process.
- Explain ICL (E-ICL): A mimicry strategy where the LLM is shown a few examples of (Input, Output, Explanation) triplets, where the explanation is sourced from a gold-standard method like LIME. The LLM then learns to generate similar explanations for new samples.
To handle tabular data, the framework uses serialization templates to convert numerical values into natural language strings suitable for LLM processing.
Experimental Setup
The framework was evaluated across eight datasets, including tabular data (e.g., Blood Transfusion, Credit Default, HELOC) and text data (Amazon, IMDb, and Yelp reviews). The target "black box" models included Logistic Regression (LR), multi-layer Artificial Neural Networks (ANNs), and Transformers. The performance was measured using metrics like Feature Agreement (FA), Rank Agreement (RA), and Prototypicality (PGU/PGI).
Key Results and Insights
Performance vs. Traditional Explainers
LLM-based explanations proved to be remarkably faithful. In many scenarios, ICE methods matched or exceeded the performance of specialized tools like LIME. For larger tabular datasets, LLM explanations achieved significantly better results than standard gradient-based baselines, showing a 50.13% lower PGU (better at ignoring noise) and 144.82% higher PGI (better at identifying key features).
The Relational Advantage
A critical finding was that LLMs perform better when shown "raw perturbations"—the specific change in input and the resulting change in output—rather than absolute values. This relational view allows the model to focus on the delta directly, bypassing complex mental arithmetic.
Model Scaling and Guidance
There is a clear intelligence threshold for these tasks; GPT-4 significantly outperformed GPT-3.5. Furthermore, the structured analytical process in PG-ICL led to more consistent and higher faithfulness scores than generic instructions, particularly on complex tabular data.
Limitations
Despite their success, LLM explainers face challenges:
- Alphabetical Bias: When features have similar importance, LLMs sometimes default to alphabetical ranking.
- Prompt Sensitivity: Results are highly dependent on the specific wording and serialization templates used.
- Cost: High-end LLM API calls can be more expensive and slower than local execution of traditional algorithms.
Conclusion
This research demonstrates that LLMs are capable of acting as effective post hoc explainers for other predictive models. By leveraging ICL and structured prompting, the ICE framework provides a promising new avenue for model interpretability, potentially reducing the computational overhead of traditional perturbation-based methods while maintaining high explanation faithfulness.
Summary updated