Compression-based Network Interpretability Schemes

Jonathan Warrell, Hussein Mohsen, Mark GersteinbioRxiv 2020 · DOI

TL;DR

This paper introduces "interpretability schemes" that use model compression to extract meaningful feature and data groupings from trained deep learning models. They demonstrate that generalization error is a good indicator of interpretability quality and propose PAC-Bayes bounds that can guide the selection of optimal compression levels for better insights.

What Problem Does This Paper Address?

Deep learning models achieve impressive performance but often act as "black boxes," making it difficult to understand why they make certain predictions. This lack of interpretability hinders trust, knowledge discovery, and the generation of new hypotheses. While model compression is often used for efficiency, its potential for extracting interpretable information has not been fully explored. The authors aim to bridge this gap by investigating how compression can be leveraged for interpretability, specifically asking: how much compression is optimal for extracting meaningful insights, and can compression be combined with other criteria for selecting the best interpretation?

Methods

The paper proposes a general framework for "interpretability schemes" consisting of three parts:

  1. Model Transformation/Compression: Applying compression techniques to the trained network.
  2. Information Extraction: Deriving interpretable information from the compressed model.
  3. Interpretable Model Selection: Using a scoring mechanism (based on PAC-Bayes generalization bounds) to select the best compressed model for interpretation.

Two novel, orthogonal compression-based interpretability schemes are introduced:

  • Rank Projection Trees (RPTs): This method uses structured sparsification to extract nested groups of features that might have joint interactions. It builds a tree structure that maps to the neural network, prioritizing inputs and feature sets based on an output-dependent ranking function.
  • Cascaded Network Decomposition (CND): This scheme splits a network into a cascade of simpler subnetworks. Each stage focuses on a subset of training instances with similar characteristics, effectively grouping data points with similar "difficulty" levels. This approach is akin to boosting.

The authors also derive PAC-Bayes generalization bounds for these schemes, which can serve as proxy indicators for interpretability quality and aid in selecting optimal compression. They further extend this framework to incorporate prior knowledge, creating a modified MDL bound.

Datasets

The proposed methods are evaluated on two real-world biological datasets:

  • PanCancer Analysis of Whole Genomes (PCAWG): Used for cancer genomics tasks, specifically predicting co-occurrence of germline and somatic mutations.
  • PsychENCODE: Used for psychiatric genomics tasks, classifying controls versus schizophrenia, bipolar disorder, and autism subjects.

Key Insights and Ideas

  • Compression as a Gateway to Interpretation: The core idea is that compressing a neural network can reveal underlying structure and relationships that are otherwise hidden.
  • Orthogonal Interpretations: RPTs focus on feature groupings (e.g., gene interactions), while CND focuses on data instance groupings (e.g., patient subtypes), offering complementary views of the model's decision-making process.
  • Generalization is Key to Interpretability: The paper demonstrates a strong link between a network's generalization error and the quality of the extracted interpretations. Models that generalize better tend to yield more meaningful insights.
  • PAC-Bayes Bounds as Selection Criteria: The derived PAC-Bayes bounds are not just theoretical tools; they are shown to be practical indicators for selecting the optimal level of compression for interpretability.
  • Flexibility of the Framework: The PAC-Bayes framework can be extended to incorporate prior domain knowledge, leading to more informed model selection.

Key Results

  • Meaningful Groupings: Both RPT and CND successfully extracted semantically meaningful gene and subject groupings from the cancer and psychiatric genomics datasets.
  • RPT for Gene Prioritization: RPT was able to identify biologically important genes and gene sets, outperforming a randomized ranking approach.
  • CND for Patient Subtyping: CND effectively extracted groupings of patients that corresponded to semantic categories like gender, age, ethnicity, and disease subtypes.
  • Predictive Power of Bounds: The MDL generalization bounds were found to be significantly correlated with semantic enrichment, suggesting they can predict interpretability quality.
  • Modified Bound Improvement: A modified MDL bound incorporating data-dependent priors showed an improved correlation with semantic enrichment compared to the standard MDL bound.

Limitations and Future Work

  • Optimal Compression Level: The paper highlights the open question of how much compression is optimal for extracting interpretable information, suggesting this is a key area for further research.
  • Generalizability Across Domains: While the schemes were tested on genomics data, the authors suggest their applicability to other domains like image processing and language, but note that optimal bounds might vary.
  • Incorporating Prior Structure: Future work could extend the framework to explicitly handle cases where prior biological structure (e.g., gene regulatory networks) is already encoded into network models.
  • Restricted Model Classes: The current schemes don't explicitly constrain the resulting model class beyond sparsity. Future work could explore embedding constraints for targeted interpretation (e.g., decision trees).

Summary updated

Abstract

Abstract Deep learning methods have achieved state-of-the-art performance in many domains of artificial intelligence, but are typically hard to interpret. Network interpretation is important for multiple reasons, including knowledge discovery, hypothesis generation, fairness and establishing trust. Model transformations provide a general approach to interpreting a trained network post-hoc: the network is approximated by a model, which is typically compressed, whose structure can be more easily interpreted in some way (we call such approaches interpretability schemes ). However, the relationship between compression and interpretation has not been fully explored: How much should a network be compressed for optimal extraction of interpretable information? Should compression be combined with other criteria when selecting model transformations? We investigate these issues using two different compression-based schemes, which aim to extract orthogonal kinds of information, pertaining to feature and data instance-based groupings respectively. The first ( rank projection trees ) uses a structured sparsification method such that nested groups of features can be extracted having potential joint interactions. The second ( cascaded network decomposition ) splits a network into a cascade of simpler networks, allowing groups of training instances with similar characteristics to be extracted at each stage of the cascade. We use predictive tasks in cancer and psychiatric genomics to assess the ability of these approaches to extract informative feature and data-point groupings from trained networks. We show that the generalization error of a network provides an indicator of the quality of the information extracted; further we derive PAC-Bayes generalization bounds for both schemes, which we show can be used as proxy indicators, and can thus provide a criterion for selecting the optimal compression. Finally, we show that the PAC-Bayes framework can be naturally modified to incorporate additional criteria alongside compression, such as prior knowledge based on previous models, which can enhance interpretable model selection.
BibTeX
@misc{warrell2020compression,
  author = {Jonathan Warrell and Hussein Mohsen and Mark Gerstein},
  title = {{Compression-based Network Interpretability Schemes}},
  year = {2020},
  doi = {10.1101/2020.10.27.358226},
  howpublished = {bioRxiv},
}
Figure