Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation

Raymond Li, Wen Xiao, Linzi Xing, Lanjun Wang, Gabriel Murray, Giuseppe Carenini

Introduction

Transformer-based models have revolutionized Natural Language Processing (NLP), yet they remain largely "black boxes" with high computational costs due to the quadratic scaling of self-attention. While some research focuses on visualizing these models to understand their behavior, and others aim to improve efficiency through fixed attention patterns, a gap remains in bridging human interpretability with architectural optimization.

This paper introduces a novel human-in-the-loop pipeline that identifies interpretable attention patterns and "injects" them back into the model architecture. By guiding models to use human-validated logic—such as attending to matching tokens or staying within sentence boundaries—the researchers achieve simultaneous improvements in task accuracy, interpretability, and computational efficiency.

The Human-in-the-Loop Pipeline

The authors propose a three-step methodology to move from a standard "black box" Transformer to a "glass box" model guided by human logic.

Overview of the Human-in-the-Loop Pipeline: (A) Trained Model, (B) Pattern Discovery, and (C) Pattern Injection.
  1. Discovery: Using a gradient-based method to rank the importance of attention heads, humans inspect the top-performing heads via a visual interface. They look for patterns that can be described as logical predicates (e.g., "Does token A match token B?").
  2. Evaluation: Candidate patterns are quantitatively tested against a validation set using the Global Relevance (GR) metric. This ensures the patterns are consistent across the dataset and not just artifacts of specific examples.
  3. Injection: Validated patterns are integrated into the model through Fixing (replacing learned weights with a fixed matrix) or Masking (nudging the model toward specific behaviors while allowing for further learning).

Interpretable Attention Patterns

Through the discovery process, three primary interpretable patterns emerged as highly effective for summarization and segmentation tasks:

  • Matching Token: Attending to other occurrences of the same word, which helps identify key terms.
  • Intra-Sentence: Restricting attention to tokens within the same sentence to capture local context.
  • Positional: Attending to immediate preceding or following tokens.
The discovery process identifies key patterns like Matching Token and Intra-Sentence by visualizing how important heads attend to specific tokens.

The synergy between "Matching Token" and "Intra-Sentence" patterns was particularly notable, effectively mimicking a neural version of TF-IDF by identifying important terms within their local context.

Key Results and Insights

The study evaluated this pipeline on extractive summarization (CNN/DM, NYT-50) and topic segmentation (WikiSection, Wiki-727K).

  • Performance Gains: Pattern-infused models consistently outperformed vanilla Transformers. In summarization, models showed an average 15% improvement over baselines when normalized against oracle scores.
  • Efficiency through Sparsity: By forcing the model to use specific patterns, the authors could "sparse out" the attention matrices. Remarkably, models outperformed dense vanilla Transformers even when forced to ignore 86% of potential connections.
  • Generalizability: Patterns discovered on one dataset transferred effectively to others, suggesting they capture fundamental linguistic aspects of the task rather than dataset-specific noise.
  • Low Human Overhead: The discovery process is efficient, requiring less than one hour of human interaction per task.

Conclusion

This research demonstrates that interpretability does not have to come at the cost of performance. By leveraging human-guided patterns, models can become more accurate, more efficient, and more transparent. Future work will explore the applicability of these patterns to more complex tasks like abstractive summarization and the potential for automating the discovery phase using predefined linguistic predicates.

Summary updated

Abstract

The transformer multi-head self-attention mechanism has been thoroughly investigated recently. On one hand, researchers are interested in understanding why and how transformers work. On the other hand, they propose new attention augmentation methods to make transformers more accurate, efficient and interpretable. In this paper, we synergize these two lines of research in a human-in-the-loop pipeline to first find important task-specific attention patterns. Then those patterns are applied, not only to the original model, but also to smaller models, as a human-guided knowledge distillation process. The benefits of our pipeline are demonstrated in a case study with the extractive summarization task. After finding three meaningful attention patterns in the popular BERTSum model, experiments indicate that when we inject such patterns, both the original and the smaller model show improvements in performance and arguably interpretability.
BibTeX
@inproceedings{li2022human,
  author = {Raymond Li and Wen Xiao and Linzi Xing and Lanjun Wang and Gabriel Murray and Giuseppe Carenini},
  title = {{Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.694},
}
Figure