Abstract
Pretrained transformers achieve the state of the art across tasks in natural language processing, motivating researchers to investigate their inner mechanisms. One common direction is to understand what features are important for prediction. In this paper, we apply information bottlenecks to analyze the attribution of each feature for prediction on a black-box model. We use BERT as the example and evaluate our approach both quantitatively and qualitatively. We show the effectiveness of our method in terms of attribution and the ability to provide insight into how information flows through layers. We demonstrate that our technique outperforms two competitive methods in degradation tests on four datasets.
Introduction
Deep learning models, particularly Transformers like BERT, have revolutionized NLP but remain notoriously difficult to interpret. Understanding why a model makes a specific prediction—known as prediction attribution—is crucial for building trust and debugging model behavior. Identifying which input tokens (e.g., specific words) are most influential is a primary goal of interpretability research.
Existing attribution methods face significant hurdles. Gradient-based approaches like Integrated Gradients (IG) can suffer from noisy or vanishing gradients that fail to capture the full scope of information used by the model. Model-agnostic methods like LIME rely on local linear approximations that may not faithfully reflect the model's complex internal logic. This work proposes a theoretically grounded approach using Information Bottlenecks (IB) to provide deeper insights into Transformer decision-making.
Methodology: Information Bottleneck for Attribution (IBA)
The proposed method, Information Bottleneck for Attribution (IBA), identifies important features by strategically restricting information flow within the network. Unlike standard training, this optimization happens at the instance level for specific inputs.
- Noise Injection: A "bottleneck" is inserted after a chosen layer, adding Gaussian noise to the internal representations.
- Mask Optimization: The method optimizes a mask ($\alpha$) to determine the minimum amount of information (minimizing mutual information with the input) required to maintain the model's original prediction.
- Attribution Scoring: Features that must remain clear (low noise) to preserve prediction accuracy are identified as highly important. The attribution score is calculated using the Kullback–Leibler (KL) divergence, measuring the information "kept" for each token.
Experimental Setup
The researchers evaluated IBA using BERT-base on four diverse NLP benchmarks:
- IMDB: Sentiment analysis of movie reviews.
- MNLI (Matched/Mismatched): Multi-Genre Natural Language Inference (entailment vs. contradiction).
- AG News: Topic classification for news articles.
- RTE: Recognizing Textual Entailment from the GLUE benchmark.
Results and Discussion
The effectiveness of IBA was measured using degradation tests. In these tests, tokens are removed in order of their attributed importance; a more accurate attribution method will cause a sharper drop in model confidence as important tokens are removed.
Quantitative Performance
As shown in the degradation curves above, IBA consistently outperforms or matches competitive methods like Integrated Gradients (IG) and LIME across all tested datasets. For instance, on the IMDB dataset, removing the top 11% of tokens identified by IBA resulted in a probability drop of 0.229, significantly higher than the drops seen for IG (0.090) and LIME (0.038).
Information Flow and Layer Sensitivity
A key insight from this research is the identification of "decision-making hubs" within the Transformer architecture. Attribution was found to be most effective when the bottleneck was placed at layers 8 or 9 of BERT. This suggests that these middle-to-late layers are where task-specific information "crystallizes" before final refinement in the output layers. Early layers (1-3) appear to contain representations that are too raw for effective attribution.
Conclusion
IBA provides a robust, theoretically grounded framework for Transformer interpretability. By treating attribution as a constrained resource problem—determining the minimum information required for a correct answer—it avoids common pitfalls of gradient-based methods. While IBA is more computationally intensive than simpler methods, its superior accuracy and ability to map internal information flow make it a valuable tool for understanding complex models. Future work will explore extending this technique to generative tasks and optimizing its computational efficiency.
Summary updated