Human Interpretation and Exploitation of Self-attention Patterns in Transformers: A Case Study in Extractive Summarization

Raymond Li, Wen Xiao, Lanjun Wang, Giuseppe Carenini

Abstract

Transformer-based models have set new benchmarks in NLP but suffer from interpretability issues and high computational costs due to redundant self-attention mechanisms. This paper presents a human-in-the-loop pipeline designed to identify interpretable attention patterns and "inject" them back into the model. By leveraging human intuition to discover task-relevant heuristics—such as matching tokens and sentence boundaries—the proposed method achieves up to a 15% performance improvement in extractive summarization and topic segmentation. Furthermore, the approach enables significant model sparsification, removing up to 86% of attention connections while maintaining superior performance over dense baselines.

Introduction

The multi-head self-attention mechanism is the cornerstone of modern NLP architectures like BERT. However, these models are often viewed as "black boxes" with attention patterns that are frequently redundant or noisy. Research typically diverges into two paths: interpretability (post-hoc analysis of learned weights) and efficiency (imposing fixed sparse patterns). This study bridges the gap by using human-guided discovery to find high-quality attention patterns and then enforcing these patterns to improve both model performance and efficiency.

The Human-in-the-Loop Pipeline

The authors propose a generic three-step framework to transition from a standard trained model to a pattern-infused, efficient architecture.

The end-to-end pipeline for pattern discovery and injection.

1. Pattern Discovery

The discovery phase begins by estimating head importance using gradient-based Taylor expansion to identify which attention heads contribute most to the final task. A human expert then visually inspects these critical heads using visualization tools (e.g., T3-vis) to identify recurring behaviors. Finally, these patterns are formalized as mathematical predicates and evaluated for "Global Relevance" (GR) across a validation set to ensure statistical significance.

2. Pattern Injection

Once validated, patterns are integrated into the model through two primary mechanisms:

  • Fixing: The learned attention matrix is replaced with a hard-coded, constant matrix representing the pattern.
  • Masking: A binary mask is applied to the learned attention, pruning connections that do not align with the discovered pattern.

3. Projected Attention Layers (PALs)

To adapt pre-trained models without full retraining, the authors utilize PALs—small, parallel modules that inject patterns into the existing BERT architecture, preserving the underlying knowledge while guiding the attention flow.

Experimental Setup

The generalizability of the pipeline was tested across two distinct document-level tasks:

  • Extractive Summarization: Evaluated on the CNN/Daily Mail (CNN/DM) and New York Times (NYT-50) datasets.
  • Topic Segmentation: Evaluated on WikiSection and Wiki-727K.

Key Insights: Universal Attention Patterns

Through human inspection, three specific patterns were identified as universally beneficial across tasks:

Visual representation of the four primary attention patterns evaluated.
  1. Matching Token: Attending to other occurrences of the same word, which helps the model identify and link key entities.
  2. Intra-Sentence/Context: Restricting attention to words within the same sentence or segment to maintain local coherence.
  3. Positional: Attending to the immediate previous (-1) or next (+1) token to capture local syntactic structures.

A critical finding was the synergy between these patterns. While positional attention alone sometimes degraded performance, its combination with "matching token" and "intra-sentence" patterns provided a substantial boost, suggesting that models require a balance of local syntax and global semantic anchors.

Results and Discussion

The pattern-infused models consistently outperformed vanilla Transformers. In summarization tasks, the models saw a 15% improvement in normalized ROUGE scores. In topic segmentation, F1-scores improved by approximately 12%.

Beyond performance, the method offers significant efficiency gains. The authors achieved a sparsity ratio of 0.86, meaning 86% of attention connections were pruned. Despite this massive reduction in complexity, the sparse, pattern-guided models still outperformed their dense counterparts. Furthermore, patterns discovered on one dataset (e.g., CNN/DM) generalized well to others (e.g., NYT-50), indicating the robustness of human-identified heuristics.

Conclusion

This work demonstrates that human intuition can effectively guide Transformer models toward more interpretable and efficient configurations. By identifying and injecting task-specific attention patterns, researchers can create models that are "interpretable by design" while simultaneously reducing computational overhead. Future work will explore the automation of the discovery phase and the application of these techniques to generative tasks and multi-lingual contexts.

Summary updated

BibTeX
@inproceedings{li2022human,
  author = {Raymond Li and Wen Xiao and Lanjun Wang and Giuseppe Carenini},
  title = {{Human Interpretation and Exploitation of Self-attention Patterns in Transformers: A Case Study in Extractive Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.694},
}
Figure