TL;DR
This paper introduces SLED, a novel framework that generates natural language explanations for why two machine learning models disagree or agree on their predictions. By learning synthetic "divergent" and "convergent" samples and then using a large language model (LLM) to explain these patterns, SLED provides more faithful and understandable insights into model behavior than existing methods, even with limited access to training data.
What Problem Does This Paper Address?
As AI models become more prevalent in critical applications, understanding why they make certain decisions is crucial for trust, debugging, and responsible deployment. While methods exist to explain individual model predictions, explaining the differences between two models is a more complex challenge. Current approaches often require full access to training data, don't produce human-readable explanations, or focus on input transformations rather than the underlying decision logic. This paper tackles the problem of generating faithful, natural language explanations that pinpoint where and how two models converge or diverge in their predictions, especially in scenarios with limited access to the original training data.
Methods
SLED employs a two-stage approach:
- Synthetic Sample Learning: This is the core innovation. SLED uses gradient-based optimization to generate synthetic input samples that specifically highlight the differences (divergent samples) and similarities (convergent samples) in how two models, $f_1$ and $f_2$, make predictions.
- For structured tasks, a custom loss function is optimized to encourage divergence (by maximizing cross-entropy between model outputs) or convergence (by minimizing it), while a regularization term ensures these synthetic samples stay close to the original data distribution.
- For text-based tasks, an additional step is needed to bridge the gap between model embedding spaces. An affine transformation is learned to map embeddings from $f_1$'s space to $f_2$'s. The synthetic samples are then generated as embeddings and mapped back to text using an LLM-generated corpus.
- Explanation Generation: The learned synthetic divergent and convergent samples are then fed to an "Explainer LLM." This LLM is prompted to generate a concise, natural language explanation that describes the general patterns and characteristics distinguishing the divergent samples from the convergent ones. The goal is to create explanations that are abstract, generalizable, and human-understandable, rather than just describing the synthetic samples themselves.
Datasets
The paper evaluates SLED on a variety of tasks:
- Text-based Classification Tasks:
- Tweet hate-speech detection (Davidson et al., 2017)
- Multi-class sentiment analysis (Parvez, 2023)
- IMDB Movie Review polarity prediction (Maas et al., 2011)
- Structured Classification Tasks:
- Wine Quality
- MAGIC Gamma Telescope
- Rice
- Bank Note
- Adult
- Bank Marketing
- Car Evaluation
- Tic Tac Toe
- Nursery
- Balance Scale
The paper does not introduce new benchmarks but uses established datasets for evaluation.
Key Insights and Ideas
- Synthetic Samples as Explanatory Tools: The most significant insight is the effectiveness of generating targeted synthetic samples to explicitly reveal model disagreements and agreements. This bypasses the need for extensive real-world data and allows for precise control over the learning process.
- LLMs for Abstract Explanations: Leveraging LLMs to interpret these synthetic samples and generate abstract, generalizable explanations is key. This moves beyond simply listing features of divergent examples to articulating the underlying behavioral patterns.
- Regularization is Crucial: The regularization loss is vital for ensuring that synthetic samples remain representative of the original data distribution. Without it, SLED tends to generate edge-case examples, leading to less faithful explanations.
- Sample Efficiency: SLED is surprisingly sample-efficient. The paper shows that even with as few as 5-10 synthetic samples, high faithfulness can be achieved, making the framework practical.
- Robustness to LLM Choice: The quality of explanations is relatively robust to the choice of the explainer LLM, meaning smaller, more accessible models can still produce good results.
Key Results
- Faithfulness: SLED significantly outperforms baselines (MaNtLE-PF, MaNtLE, LIME, Anchors) on structured tasks, achieving 24-48% higher faithfulness. On text tasks, it is 18% more faithful than an LLM baseline with full training data access.
- Simulatability: SLED explanations also show strong simulatability, meaning they enable users to accurately predict model behavior on unseen data. SLED scores are 22% higher than top baselines on structured tasks and 10% higher than the LLM baseline on text tasks.
- Human Evaluation: User studies reveal that SLED explanations achieve a real-world simulatability score of 63.5%. Participants found the explanations understandable (3.86/5), informative (4.61/5), and useful (3.87/5).
Limitations and Future Work
- Differentiability Requirement: SLED relies on gradient-based optimization, meaning it is only applicable to differentiable ML models.
- LLM Dependency: The quality of explanations is inherently tied to the capabilities of the chosen LLM, both for generating synthetic text and for producing the final explanation.
- Task Applicability: The current framework is designed for classification tasks and has not been explored for generative tasks.
- Out-of-Distribution Inputs: If synthetic samples are too similar to the training data, explanations might not capture behavior on truly out-of-distribution inputs.
- Demographic Diversity: The human evaluation was limited in demographic diversity.
Future work includes extending SLED to multi-model tasks, enabling interactive settings for user-driven "why" questions, and exploring safeguards against potential misuse of the generated explanations.
Summary updated