Abstract
One principal approach for illuminating a black-box neural network is feature attribution, i.e. identifying the importance of input features for the network's prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.
Introduction
Deep learning models are notoriously "black boxes," making it difficult to understand the reasoning behind their predictions. Feature attribution aims to solve this by highlighting which parts of an input—such as specific pixels in an image or words in a sentence—were most important for a decision.
Existing methods based on the Information Bottleneck (IB) principle, like the original IBA, typically inject noise into latent (hidden) layers to identify essential information. However, these methods face a "blurriness" problem: to visualize results, latent maps must be interpolated back to the input resolution, leading to a significant loss of detail. Furthermore, these methods often rely on spatial correspondences found in Convolutional Neural Networks (CNNs), limiting their applicability to other architectures like Recurrent Neural Networks (RNNs).
Methodology: InputIBA
The researchers developed InputIBA, a novel method that moves the information bottleneck from the hidden layers directly to the input domain. This shift allows for sharper, more accurate, and architecture-agnostic explanations.
Because computing predictive information directly at the input is mathematically intractable, InputIBA employs a multi-step optimization process:
- Latent Target Identification: The method first identifies an optimal bottleneck at a deep layer where information is most concentrated.
- Generative Matching: A generative adversarial optimization scheme finds an input-level bottleneck ($\lambda_G^*$) that induces a feature distribution matching the deep latent bottleneck.
- Fine-Grained Optimization: Using the input mask as a prior, a final optimization produces a sparse mask ($\Lambda^*$) that maximizes mutual information with the model's output.
Experimental Evaluation
InputIBA was evaluated across two domains: Computer Vision (ImageNet with VGG-16) and Natural Language Processing (IMDB sentiment analysis with a 4-layer LSTM).
Key Results
- Visual Precision: Qualitatively, InputIBA produces much sharper maps than baselines. While GradCAM might highlight a whole object, InputIBA pinpoints specific discriminative features like a snout or ears.
- Quantitative Superiority: On the Insertion/Deletion metric (measuring how quickly prediction confidence drops as pixels are removed), InputIBA outperformed all baselines. On ImageNet, it achieved a Deletion AUC of 0.045, significantly better than the original IBA (0.090) and GradCAM (0.133).
- Architecture Agnosticism: Unlike previous IB methods, InputIBA successfully identified key sentiment words (e.g., "best," "solid") in RNN-based NLP tasks, where the original IBA failed to produce meaningful explanations.
- Localization Precision: The authors introduced the Effective Heat Ratio (EHR) to better capture how precisely an attribution method pinpoints relevant objects. InputIBA scored 0.476, nearly doubling the score of GradCAM (0.283).
Discussion and Insights
A striking observation from this research is the "Axiomatic Paradox": mathematically elegant "axiomatic" methods (like Integrated Gradients) often perform poorly on empirical feature importance tests. They may look fine-grained to humans but do not always highlight the features the model actually relies on. InputIBA avoids the overestimation of information common in early-layer IB methods by using deep layers to guide the input-level explanation.
Limitations
InputIBA is more complex to implement and computationally more expensive than "plug-and-play" methods like GradCAM, as it requires an additional optimization step for every explanation. Future work will focus on streamlining this process for real-time applications and large-scale auditing.
Summary updated