Introduction
Transformer-based models have revolutionized Natural Language Processing (NLP), yet they remain largely "black boxes" with high computational costs due to the quadratic scaling of self-attention. While some research focuses on visualizing these models to understand their behavior, and others aim to improve efficiency through fixed attention patterns, a gap remains in bridging human interpretability with architectural optimization.
This paper introduces a novel human-in-the-loop pipeline that identifies interpretable attention patterns and "injects" them back into the model architecture. By guiding models to use human-validated logic—such as attending to matching tokens or staying within sentence boundaries—the researchers achieve simultaneous improvements in task accuracy, interpretability, and computational efficiency.
The Human-in-the-Loop Pipeline
The authors propose a three-step methodology to move from a standard "black box" Transformer to a "glass box" model guided by human logic.
- Discovery: Using a gradient-based method to rank the importance of attention heads, humans inspect the top-performing heads via a visual interface. They look for patterns that can be described as logical predicates (e.g., "Does token A match token B?").
- Evaluation: Candidate patterns are quantitatively tested against a validation set using the Global Relevance (GR) metric. This ensures the patterns are consistent across the dataset and not just artifacts of specific examples.
- Injection: Validated patterns are integrated into the model through Fixing (replacing learned weights with a fixed matrix) or Masking (nudging the model toward specific behaviors while allowing for further learning).
Interpretable Attention Patterns
Through the discovery process, three primary interpretable patterns emerged as highly effective for summarization and segmentation tasks:
- Matching Token: Attending to other occurrences of the same word, which helps identify key terms.
- Intra-Sentence: Restricting attention to tokens within the same sentence to capture local context.
- Positional: Attending to immediate preceding or following tokens.
The synergy between "Matching Token" and "Intra-Sentence" patterns was particularly notable, effectively mimicking a neural version of TF-IDF by identifying important terms within their local context.
Key Results and Insights
The study evaluated this pipeline on extractive summarization (CNN/DM, NYT-50) and topic segmentation (WikiSection, Wiki-727K).
- Performance Gains: Pattern-infused models consistently outperformed vanilla Transformers. In summarization, models showed an average 15% improvement over baselines when normalized against oracle scores.
- Efficiency through Sparsity: By forcing the model to use specific patterns, the authors could "sparse out" the attention matrices. Remarkably, models outperformed dense vanilla Transformers even when forced to ignore 86% of potential connections.
- Generalizability: Patterns discovered on one dataset transferred effectively to others, suggesting they capture fundamental linguistic aspects of the task rather than dataset-specific noise.
- Low Human Overhead: The discovery process is efficient, requiring less than one hour of human interaction per task.
Conclusion
This research demonstrates that interpretability does not have to come at the cost of performance. By leveraging human-guided patterns, models can become more accurate, more efficient, and more transparent. Future work will explore the applicability of these patterns to more complex tasks like abstractive summarization and the potential for automating the discovery phase using predefined linguistic predicates.
Summary updated