Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information

Yang Zhang, Ashkan Khakzar, Yawei Li, Azade Farshad, Seong Tae Kim, Nassir Navab

Abstract

One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network's prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.

Introduction

Deep learning models are notoriously "black boxes," making it difficult to understand the reasoning behind their predictions. Feature attribution aims to solve this by highlighting which parts of an input—such as specific pixels in an image or words in a sentence—were most important for a decision.

Existing methods based on the Information Bottleneck (IB) principle, like the original IBA, typically inject noise into latent (hidden) layers to identify essential information. However, these methods face a "blurriness" problem: to visualize results, latent maps must be interpolated back to the input resolution, leading to a significant loss of detail. Furthermore, these methods often rely on spatial correspondences found in Convolutional Neural Networks (CNNs), limiting their applicability to other architectures like Recurrent Neural Networks (RNNs).

Methodology: InputIBA

The researchers developed InputIBA, a novel method that moves the information bottleneck from the hidden layers directly to the input domain. This shift allows for sharper, more accurate, and architecture-agnostic explanations.

Because computing predictive information directly at the input is mathematically intractable, InputIBA employs a multi-step optimization process:

  1. Latent Target Identification: The method first identifies an optimal bottleneck at a deep layer where information is most concentrated.
  2. Generative Matching: A generative adversarial optimization scheme finds an input-level bottleneck ($\lambda_G^*$) that induces a feature distribution matching the deep latent bottleneck.
  3. Fine-Grained Optimization: Using the input mask as a prior, a final optimization produces a sparse mask ($\Lambda^*$) that maximizes mutual information with the model's output.
The progression from coarse latent-layer IBA maps to the final fine-grained InputIBA mask ($\Lambda^*$).

Experimental Evaluation

InputIBA was evaluated across two domains: Computer Vision (ImageNet with VGG-16) and Natural Language Processing (IMDB sentiment analysis with a 4-layer LSTM).

Key Results

  • Visual Precision: Qualitatively, InputIBA produces much sharper maps than baselines. While GradCAM might highlight a whole object, InputIBA pinpoints specific discriminative features like a snout or ears.
  • Quantitative Superiority: On the Insertion/Deletion metric (measuring how quickly prediction confidence drops as pixels are removed), InputIBA outperformed all baselines. On ImageNet, it achieved a Deletion AUC of 0.045, significantly better than the original IBA (0.090) and GradCAM (0.133).
  • Architecture Agnosticism: Unlike previous IB methods, InputIBA successfully identified key sentiment words (e.g., "best," "solid") in RNN-based NLP tasks, where the original IBA failed to produce meaningful explanations.
  • Localization Precision: The authors introduced the Effective Heat Ratio (EHR) to better capture how precisely an attribution method pinpoints relevant objects. InputIBA scored 0.476, nearly doubling the score of GradCAM (0.283).

Discussion and Insights

A striking observation from this research is the "Axiomatic Paradox": mathematically elegant "axiomatic" methods (like Integrated Gradients) often perform poorly on empirical feature importance tests. They may look fine-grained to humans but do not always highlight the features the model actually relies on. InputIBA avoids the overestimation of information common in early-layer IB methods by using deep layers to guide the input-level explanation.

Limitations

InputIBA is more complex to implement and computationally more expensive than "plug-and-play" methods like GradCAM, as it requires an additional optimization step for every explanation. Future work will focus on streamlining this process for real-time applications and large-scale auditing.

Summary updated

Abstract

One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network's prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.
BibTeX
@inproceedings{zhang2021fine,
  author = {Yang Zhang and Ashkan Khakzar and Yawei Li and Azade Farshad and Seong Tae Kim and Nassir Navab},
  title = {{Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=vIUTwImhfz},
}
Figure