Abstract
As machine learning models grow in complexity, their interpretability becomes a critical bottleneck for trust and deployment in high-stakes fields like medicine and genomics. This paper introduces a model-agnostic framework designed to interpret "black-box" models by identifying important features and their interactions at varying levels of granularity. By combining hierarchical feature grouping with rigorous statistical hypothesis testing, the method pinpoints the "right resolution"—the most specific level of detail—needed to understand model decisions while strictly controlling the false discovery rate (FDR).
Introduction
Modern predictive models, such as Deep Neural Networks and Random Forests, often rely on thousands of input features. Traditional interpretability methods typically focus on these "base features" in isolation. However, in domains like Electronic Health Records (EHR), individual data points (e.g., a rare diagnosis code) may lack the statistical power to show significance, whereas a broader category (e.g., "Respiratory Diseases") might be a powerful predictor. Conversely, a model might rely on a highly specific genetic marker that would be obscured if grouped too broadly.
The challenge lies in finding the optimal resolution for interpretation: identifying when a model relies on a broad category versus a specific data point, and doing so without being misled by statistical noise.
Methodology
The proposed framework is model-agnostic and built upon five core pillars:
- Feature Hierarchies: Features are organized into a tree structure where internal nodes represent groups (e.g., "Heart Diseases") and leaves represent base features (e.g., "Acute Myocardial Infarction").
- Perturbation-Based Importance: Feature importance is measured by "erasing" or permuting a node's values in the test data and observing the subsequent increase in model loss.
- Hierarchical Hypothesis Testing: The authors employ the Wilcoxon matched-pairs signed-rank test to determine if a change in loss is statistically significant. Testing proceeds top-down; children are only evaluated if their parent node is found to be significant.
- FDR Control: To manage the multiple comparisons problem inherent in testing thousands of groups, a hierarchical False Discovery Rate (FDR) control algorithm is implemented to ensure statistical reliability.
- Interaction Detection: The method identifies non-additive interactions where perturbing two features together impacts the model significantly differently than the sum of their individual perturbations.
Experimental Evaluation
The researchers validated their approach across three distinct domains:
- Synthetic Data: Used to confirm that the FDR control maintains the target threshold (0.05) while preserving high power to detect true signals.
- Viral Genomics (HSV-1): Analyzing 547 genetic features of the Herpes Simplex Virus mapped to disease severity in mice using a Random Forest model.
- Electronic Health Records (Asthma): Predicting asthma exacerbations for over 28,000 patients using an LSTM neural network fed with thousands of ICD-9 codes, medications, and demographic features.
Key Results and Discussion
The "Outer Node" Concept
A central output of this method is the identification of "outer nodes"—the finest level of the hierarchy where feature importance remains statistically significant. This identifies exactly where the model's predictive knowledge resides. In the HSV-1 task, the method successfully narrowed 1,093 candidate nodes down to just 40 outer nodes, highlighting both known pathogenic regions and novel areas for biological investigation.
Resolution and Clinical Logic
In the asthma study, the hierarchical approach revealed that the model's logic operates at uneven resolutions. While it might care about broad categories like "Metabolic Diseases," it hones in on very specific sub-types for "Respiratory Diseases." Specifically, the method traced importance through the hierarchy to the "Asthma" code (ICD 493.90).
Statistical Rigor vs. Ranking
Unlike tools that merely rank features, this framework provides p-values to distinguish true contributors from random variance. Interestingly, the study found that even if all specific features were erased, the mere frequency of hospital encounters provided a baseline predictive signal (AUROC 0.537), demonstrating the method's ability to isolate different types of model knowledge.
Conclusion and Future Work
While effective, the current framework assumes features are organized in a strict tree structure. Future work aims to extend this to overlapping groups (directed acyclic graphs) and to explore the "right resolution" in the temporal domain—determining, for instance, whether a model relies on a patient's history from the last week or the last year. Additionally, the search for interactions is currently limited to features that are individually significant, which may miss "pure" interactions where two features are only important when combined.
Summary updated