TL;DR
This paper introduces Mahé, a novel framework for explaining how complex machine learning models, particularly deep neural networks, capture interactions between features. Mahé provides both "context-dependent" explanations (specific to individual data points) and "context-free" explanations (generalizable global behaviors), outperforming existing methods in local interaction interpretation and offering a new way to understand model decision-making.
What Problem Does This Paper Address?
Modern machine learning models, especially deep neural networks, excel at learning intricate dependencies within data like text and images. However, understanding how these models capture complex relationships, particularly interactions between features (e.g., "not" and "bad" in sentiment analysis, or object relationships in images), remains a significant challenge. Existing explanation methods often focus on individual feature contributions (feature attribution) and struggle to effectively represent or explain these non-additive, multi-feature interactions, especially when their behavior varies depending on the specific data instance. This paper aims to bridge this gap by providing a unified framework for explaining both instance-specific (context-dependent) and globally consistent (context-free) interactions.
Methods
Mahé is a model-agnostic framework that offers two main types of explanations:
- Context-Dependent Explanations: Mahé first identifies interacting variables within the local vicinity of a data instance. It then uses a novel local interpretation algorithm that can capture arbitrary-order interactions, going beyond simple linear models. This is achieved by fitting a generalized additive model (GAM) with interaction terms, where the functions are learned using Multilayer Perceptrons (MLPs). This approach allows for hierarchical explanations, progressively revealing more complex interactions.
- Context-Free Explanations: To determine if a local interaction is "context-free" (i.e., consistently behaves the same way globally), Mahé tests whether modifying the local interaction's behavior affects its global influence. This involves negating a local interaction's attribution and observing if the polarity of the interaction's contribution flips across many representative data instances.
The core of Mahé's interaction detection relies on the Neural Interaction Detection (NID) framework, which efficiently identifies interactions by interpreting MLP weights.
Datasets
The paper evaluates Mahé on both synthetic and real-world datasets:
- Synthetic Datasets: Used to evaluate context-dependent explanations, these datasets are generated from known interaction functions (F1-F4) with 10 features.
- Real-World Datasets:
- DNA-CNN: Trained on MYC-DNA binding data.
- Sentiment-LSTM: Trained on the Stanford Sentiment Treebank (SST) and augmented with data from IMDB movie reviews for context-free evaluation.
- ResNet152: An image classifier pre-trained on ImageNet.
- Transformer: A machine translation model pre-trained on WMT-14 En→Fr.
Key Insights and Ideas
- Interactions are Crucial: The paper strongly suggests that interactions are not just noise but fundamental components of how complex models make predictions, and their absence or misrepresentation can lead to less accurate or trustworthy explanations.
- Context Matters for Interactions: Interactions can behave differently depending on the specific data instance. Mahé's ability to distinguish between context-dependent and context-free interactions is a key innovation.
- Hierarchical Explanations Provide Depth: By building explanations hierarchically, Mahé can reveal increasingly complex interactions, offering a more nuanced understanding than single-level explanations.
- Context-Free Interactions Offer Global Understanding: Identifying interactions that consistently behave the same way across different data instances is vital for understanding global model behavior and potentially for model editing.
- Model Agnosticism is Key: Mahé's ability to work with various complex models (MLPs, LSTMs, CNNs, Transformers) makes it broadly applicable.
Key Results
- Improved Local Interaction Interpretation: Mahé significantly outperforms state-of-the-art methods like Shap-Tree, ACD-MLP, and ACD-LSTM in both fitting and detecting interactions on synthetic data, as measured by MSE and R-precision.
- Significant Performance Gains: Including interactions in Mahé's explanations led to significant performance improvements across various models and datasets (e.g., reducing prediction error by up to 30% for DNA-CNN and Sentiment-LSTM).
- Human Preference for Interactions: In a Mechanical Turk study, human evaluators preferred Mahé's explanations that included interactions over standard LIME explanations 65% of the time.
- Successful Context-Free Explanation: Mahé successfully identified context-free interactions in Sentiment-LSTM (e.g., "(too, bad)" and "(only, worse)" consistently contributing positively to sentiment) and Transformer (e.g., "this" and "article" influencing the presence of "cet" in French translations).
- Model Editing Potential: Modifying local context-free interactions in Sentiment-LSTM and Transformer led to predictable changes in global behavior (e.g., changing sentiment polarity or removing specific translation patterns) with minimal impact on overall model accuracy.
Limitations and Future Work
- NID Limitations: While NID is efficient, there's no guarantee it always finds the "correct" interactions.
- High-Dimensional Data Challenges: Mahé, like other perturbation-based methods, faces challenges with very high-dimensional inputs (e.g., images) and the choice of perturbation distributions.
- Computational Cost: While Mahé is model-agnostic, the process of interaction detection and fitting can be computationally intensive, especially for large numbers of features. The paper suggests limiting feature dimensions to under 100 for practical runtime.
- Future Work: The authors propose making context-free interaction finding more efficient, studying how editing interactions can change model behavior, and further exploring the insights gained from Mahé's explanations in structured data. They also note that generalizing to image interactions is an ongoing research challenge.
Summary updated