Feature Importance Explanations for Temporal Black-Box Models

Akshay Sood, Mark CravenAAAI 2022 · DOI

TL;DR

This paper introduces TIME, a novel model-agnostic method for explaining temporal black-box models. TIME goes beyond existing methods by not only identifying important features and their temporal windows but also assessing the significance of feature ordering within those windows, providing a more comprehensive understanding of how these models make predictions over time.

What Problem Does This Paper Address?

Many powerful machine learning models, especially those applied to domains like healthcare, are "black boxes" – their internal workings are too complex for humans to easily understand. This lack of interpretability is a critical issue, hindering trust, debugging, and the discovery of new insights. While methods exist to explain models trained on static, tabular data, a significant gap remains for models that process sequential or time-series data. These temporal models, common in areas like patient monitoring or financial forecasting, present unique challenges because the order and timing of data points are crucial. Existing explanation methods often fail when applied to temporal data or require specific model architectures, limiting their applicability. This paper aims to bridge this gap by providing a general-purpose, model-agnostic explanation tool specifically designed for temporal models.

Methods

The core of TIME is a permutation-based approach, extended from tabular data explanations to handle temporal sequences. Here's a breakdown of its key components:

  • Model-Agnostic: TIME treats the model as a black box, meaning it doesn't need access to the model's internal architecture or parameters. It works by observing how the model's predictions change when parts of the input are perturbed.
  • Global Explanations: Instead of explaining individual predictions (local explanations), TIME aims to understand the model's behavior across an entire dataset (global explanations). This is crucial for understanding overall model logic and identifying general patterns.
  • Temporal Feature Importance: TIME uses permutations to assess the importance of individual features. For temporal data, this involves permuting entire time series or specific windows within them.
  • Window Identification: A key innovation is TIME's ability to identify the most important temporal "windows" (contiguous segments of time) within a feature's time series that the model focuses on. This is achieved through a binary search algorithm that iteratively expands and contracts potential windows based on their importance scores.
  • Ordering Importance: TIME further distinguishes itself by assessing whether the order of values within an important window matters to the model. This is done by permuting values within the identified window and observing the impact on model performance.
  • Hypothesis Testing and FDR Control: To ensure statistical rigor, TIME employs hypothesis testing and a false discovery rate (FDR) control methodology. This helps to distinguish truly important features and windows from random fluctuations and manage the problem of multiple comparisons inherent in testing many features and windows.

Datasets

The paper evaluates TIME using two types of datasets:

  • Synthetic Data: The authors generate synthetic time-series data with controlled generating processes, relevant features, and temporal properties. This allows them to precisely evaluate TIME's performance (power and FDR) against known ground truth.
  • MIMIC-III Benchmark: TIME is also applied to a real-world dataset from the MIMIC-III database, which contains critical care unit (ICU) patient data. An LSTM model trained on this data to predict in-hospital mortality is used for evaluation.

Key Insights and Ideas

  • Beyond Feature Importance: Window and Ordering Matter: The most significant insight is that for temporal models, simply knowing which features are important isn't enough. Understanding when within a feature's timeline the model is paying attention (the important window) and if the sequence of values within that window is critical provides a much richer and actionable explanation.
  • Statistical Rigor for Temporal Explanations: Incorporating hypothesis testing and FDR control is crucial for temporal explanations. This moves beyond simply reporting importance scores to providing statistically sound evidence for the identified features, windows, and ordering effects.
  • Global Explanations for Clinical Impact: The focus on global explanations is particularly relevant for high-stakes domains like healthcare. Understanding the overall logic of a model before deployment is essential for trust and safety, and TIME directly addresses this need.
  • Conciseness and Interpretability: TIME's ability to identify contiguous windows, rather than just individual timesteps, leads to more concise and interpretable explanations, making it easier for domain experts to grasp the model's reasoning.

Key Results

  • Superior Performance on Synthetic Data: On synthetic datasets, TIME significantly outperforms baseline methods in terms of average power (correctly identifying relevant features/timesteps) and FDR (controlling false positives) for both feature and timestep importance.
  • Effective Window and Ordering Detection: TIME successfully identifies important temporal windows and the significance of ordering within those windows, a capability lacking in most baseline methods.
  • Competitive Performance on Real-World Data: When applied to an LSTM model predicting in-hospital mortality from MIMIC-III data, TIME identifies a salient subset of features and windows that, when used for feature selection, result in a retrained model with an AUROC close to the original, outperforming models trained with randomly selected features.
  • Concise Explanations: TIME reports significantly fewer contiguous windows compared to baselines, leading to more interpretable explanations.

Limitations and Future Work

  • Assumption of Contiguous Windows: TIME assumes a single contiguous important window per feature. While it can gracefully identify a single window even if multiple exist, it might degrade if the model truly focuses on multiple, disjointed temporal segments.
  • Out-of-Distribution Perturbations: Like other permutation-based methods, TIME's perturbations might lead to out-of-distribution inputs, potentially inflating importance scores for correlated features. The authors suggest perturbing groups of correlated features as a future direction.
  • Regularly Sampled, Fixed-Length Sequences: The current approach assumes regularly sampled, fixed-length sequences. Future work aims to extend TIME to handle irregularly sampled data and sequences of varying lengths, as well as to consider windows aligned on absolute or relative scales (e.g., calendar dates, patient age).
  • Conditional Permutations: Exploring conditional permutations is also suggested to further refine the importance assessment.

Summary updated

BibTeX
@inproceedings{sood2022feature,
  author = {Akshay Sood and Mark Craven},
  title = {{Feature Importance Explanations for Temporal Black-Box Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20810},
}
Figure