TL;DR
This paper introduces "interpretability schemes" that use model compression to extract meaningful feature and data groupings from trained deep learning models. They demonstrate that generalization error is a good indicator of interpretability quality and propose PAC-Bayes bounds that can guide the selection of optimal compression levels for better insights.
What Problem Does This Paper Address?
Deep learning models achieve impressive performance but often act as "black boxes," making it difficult to understand why they make certain predictions. This lack of interpretability hinders trust, knowledge discovery, and the generation of new hypotheses. While model compression is often used for efficiency, its potential for extracting interpretable information has not been fully explored. The authors aim to bridge this gap by investigating how compression can be leveraged for interpretability, specifically asking: how much compression is optimal for extracting meaningful insights, and can compression be combined with other criteria for selecting the best interpretation?
Methods
The paper proposes a general framework for "interpretability schemes" consisting of three parts:
- Model Transformation/Compression: Applying compression techniques to the trained network.
- Information Extraction: Deriving interpretable information from the compressed model.
- Interpretable Model Selection: Using a scoring mechanism (based on PAC-Bayes generalization bounds) to select the best compressed model for interpretation.
Two novel, orthogonal compression-based interpretability schemes are introduced:
- Rank Projection Trees (RPTs): This method uses structured sparsification to extract nested groups of features that might have joint interactions. It builds a tree structure that maps to the neural network, prioritizing inputs and feature sets based on an output-dependent ranking function.
- Cascaded Network Decomposition (CND): This scheme splits a network into a cascade of simpler subnetworks. Each stage focuses on a subset of training instances with similar characteristics, effectively grouping data points with similar "difficulty" levels. This approach is akin to boosting.
The authors also derive PAC-Bayes generalization bounds for these schemes, which can serve as proxy indicators for interpretability quality and aid in selecting optimal compression. They further extend this framework to incorporate prior knowledge, creating a modified MDL bound.
Datasets
The proposed methods are evaluated on two real-world biological datasets:
- PanCancer Analysis of Whole Genomes (PCAWG): Used for cancer genomics tasks, specifically predicting co-occurrence of germline and somatic mutations.
- PsychENCODE: Used for psychiatric genomics tasks, classifying controls versus schizophrenia, bipolar disorder, and autism subjects.
Key Insights and Ideas
- Compression as a Gateway to Interpretation: The core idea is that compressing a neural network can reveal underlying structure and relationships that are otherwise hidden.
- Orthogonal Interpretations: RPTs focus on feature groupings (e.g., gene interactions), while CND focuses on data instance groupings (e.g., patient subtypes), offering complementary views of the model's decision-making process.
- Generalization is Key to Interpretability: The paper demonstrates a strong link between a network's generalization error and the quality of the extracted interpretations. Models that generalize better tend to yield more meaningful insights.
- PAC-Bayes Bounds as Selection Criteria: The derived PAC-Bayes bounds are not just theoretical tools; they are shown to be practical indicators for selecting the optimal level of compression for interpretability.
- Flexibility of the Framework: The PAC-Bayes framework can be extended to incorporate prior domain knowledge, leading to more informed model selection.
Key Results
- Meaningful Groupings: Both RPT and CND successfully extracted semantically meaningful gene and subject groupings from the cancer and psychiatric genomics datasets.
- RPT for Gene Prioritization: RPT was able to identify biologically important genes and gene sets, outperforming a randomized ranking approach.
- CND for Patient Subtyping: CND effectively extracted groupings of patients that corresponded to semantic categories like gender, age, ethnicity, and disease subtypes.
- Predictive Power of Bounds: The MDL generalization bounds were found to be significantly correlated with semantic enrichment, suggesting they can predict interpretability quality.
- Modified Bound Improvement: A modified MDL bound incorporating data-dependent priors showed an improved correlation with semantic enrichment compared to the standard MDL bound.
Limitations and Future Work
- Optimal Compression Level: The paper highlights the open question of how much compression is optimal for extracting interpretable information, suggesting this is a key area for further research.
- Generalizability Across Domains: While the schemes were tested on genomics data, the authors suggest their applicability to other domains like image processing and language, but note that optimal bounds might vary.
- Incorporating Prior Structure: Future work could extend the framework to explicitly handle cases where prior biological structure (e.g., gene regulatory networks) is already encoded into network models.
- Restricted Model Classes: The current schemes don't explicitly constrain the resulting model class beyond sparsity. Future work could explore embedding constraints for targeted interpretation (e.g., decision trees).
Summary updated