Understanding Learned Models by Identifying Important Features at the Right Resolution

Kyubin Lee, Akshay Sood, Mark CravenAAAI 2019 · DOI

Abstract

As machine learning models grow in complexity, their interpretability becomes a critical bottleneck for trust and deployment in high-stakes fields like medicine and genomics. This paper introduces a model-agnostic framework designed to interpret "black-box" models by identifying important features and their interactions at varying levels of granularity. By combining hierarchical feature grouping with rigorous statistical hypothesis testing, the method pinpoints the "right resolution"—the most specific level of detail—needed to understand model decisions while strictly controlling the false discovery rate (FDR).

Introduction

Modern predictive models, such as Deep Neural Networks and Random Forests, often rely on thousands of input features. Traditional interpretability methods typically focus on these "base features" in isolation. However, in domains like Electronic Health Records (EHR), individual data points (e.g., a rare diagnosis code) may lack the statistical power to show significance, whereas a broader category (e.g., "Respiratory Diseases") might be a powerful predictor. Conversely, a model might rely on a highly specific genetic marker that would be obscured if grouped too broadly.

The challenge lies in finding the optimal resolution for interpretation: identifying when a model relies on a broad category versus a specific data point, and doing so without being misled by statistical noise.

Methodology

The proposed framework is model-agnostic and built upon five core pillars:

  1. Feature Hierarchies: Features are organized into a tree structure where internal nodes represent groups (e.g., "Heart Diseases") and leaves represent base features (e.g., "Acute Myocardial Infarction").
  2. Perturbation-Based Importance: Feature importance is measured by "erasing" or permuting a node's values in the test data and observing the subsequent increase in model loss.
  3. Hierarchical Hypothesis Testing: The authors employ the Wilcoxon matched-pairs signed-rank test to determine if a change in loss is statistically significant. Testing proceeds top-down; children are only evaluated if their parent node is found to be significant.
  4. FDR Control: To manage the multiple comparisons problem inherent in testing thousands of groups, a hierarchical False Discovery Rate (FDR) control algorithm is implemented to ensure statistical reliability.
  5. Interaction Detection: The method identifies non-additive interactions where perturbing two features together impacts the model significantly differently than the sum of their individual perturbations.

Experimental Evaluation

The researchers validated their approach across three distinct domains:

  • Synthetic Data: Used to confirm that the FDR control maintains the target threshold (0.05) while preserving high power to detect true signals.
  • Viral Genomics (HSV-1): Analyzing 547 genetic features of the Herpes Simplex Virus mapped to disease severity in mice using a Random Forest model.
  • Electronic Health Records (Asthma): Predicting asthma exacerbations for over 28,000 patients using an LSTM neural network fed with thousands of ICD-9 codes, medications, and demographic features.

Key Results and Discussion

The "Outer Node" Concept

A central output of this method is the identification of "outer nodes"—the finest level of the hierarchy where feature importance remains statistically significant. This identifies exactly where the model's predictive knowledge resides. In the HSV-1 task, the method successfully narrowed 1,093 candidate nodes down to just 40 outer nodes, highlighting both known pathogenic regions and novel areas for biological investigation.

Quantitative summary of the hierarchical testing process across the HSV-1 and asthma datasets, highlighting the reduction from total nodes to significant 'outer nodes'.

Resolution and Clinical Logic

In the asthma study, the hierarchical approach revealed that the model's logic operates at uneven resolutions. While it might care about broad categories like "Metabolic Diseases," it hones in on very specific sub-types for "Respiratory Diseases." Specifically, the method traced importance through the hierarchy to the "Asthma" code (ICD 493.90).

Hierarchical feature importance for asthma exacerbation prediction. (a) High-level feature groups and their AUROC. (b) Detailed resolution within the "DIAGNOSES" subtree, identifying specific ICD-9 codes.

Statistical Rigor vs. Ranking

Unlike tools that merely rank features, this framework provides p-values to distinguish true contributors from random variance. Interestingly, the study found that even if all specific features were erased, the mere frequency of hospital encounters provided a baseline predictive signal (AUROC 0.537), demonstrating the method's ability to isolate different types of model knowledge.

Conclusion and Future Work

While effective, the current framework assumes features are organized in a strict tree structure. Future work aims to extend this to overlapping groups (directed acyclic graphs) and to explore the "right resolution" in the temporal domain—determining, for instance, whether a model relies on a patient's history from the last week or the last year. Additionally, the search for interactions is currently limited to features that are individually significant, which may miss "pure" interactions where two features are only important when combined.

Summary updated

BibTeX
@inproceedings{lee2019understanding,
  author = {Kyubin Lee and Akshay Sood and Mark Craven},
  title = {{Understanding Learned Models by Identifying Important Features at the Right Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014155},
}
Figure