Dodrio: Exploring Transformer Models with Interactive Visualization

Zijie J. Wang, Robert Turko, Duen Horng Chau

TL;DR

This paper introduces DODRIO, an interactive visualization tool designed to help researchers understand how transformer models, like BERT, learn and represent linguistic information through their attention mechanisms. By linking attention weights to syntactic and semantic knowledge, DODRIO offers novel ways to explore these complex models, providing insights into their successes and failures.

What Problem Does This Paper Address?

Large pre-trained transformer models have revolutionized Natural Language Processing (NLP), achieving state-of-the-art results across numerous tasks. A key component of these models is the multi-headed attention mechanism, which is believed to be responsible for their ability to learn and represent complex linguistic information. However, understanding how these models learn and utilize both syntactic (grammatical structure) and semantic (meaning) knowledge remains a significant challenge. This lack of interpretability hinders our ability to debug models, improve their performance, and build trust in their predictions. Existing tools often focus on visualizing attention weights in isolation, failing to connect them to crucial linguistic context.

Methods

DODRIO is an interactive, web-based visualization system that addresses this interpretability gap. Its core innovation lies in tightly integrating multiple views to connect attention weights with linguistic information:

  • Overview + Detail: DODRIO provides a high-level overview of all attention heads in a multi-layer, multi-head transformer model, allowing users to quickly identify potentially interesting heads. This overview is then linked to detailed views for in-depth exploration.
  • Linguistic Knowledge Encoding: Attention heads are visualized as circles, where color encodes their alignment with semantic (red) or syntactic (blue) information, and size represents their importance within the model. This allows for a quick assessment of a head's role.
  • Multiple Visualization Techniques: DODRIO employs various graph visualization techniques, including Dependency Views, Semantic Attention Graphs (using force, grid, and radial layouts), and Comparison Views. These are designed to represent attention weights scalably, even for longer input texts.
  • Linking and Brushing: Interactive features allow users to link elements across different views. For instance, hovering over a token in one view can highlight its associated attention patterns in another, facilitating a holistic understanding.
  • Syntactic and Semantic Integration: The tool explicitly visualizes syntactic dependencies alongside attention weights, and uses semantic saliency scores to highlight the importance of tokens.

Datasets

The paper demonstrates DODRIO's capabilities using the SST2 dataset for sentiment analysis and the Penn Treebank dataset for syntactic analysis. The tool is designed to be general and can be extended to other models and datasets.

Key Insights and Ideas

  • Attention Heads Have Specialized Roles: The visualization clearly shows that different attention heads within a transformer model specialize in capturing either semantic or syntactic information, or a combination of both. This supports the hypothesis that multi-head attention is crucial for learning diverse linguistic aspects.
  • Visualizing Linguistic Alignment is Key: By encoding semantic and syntactic alignment into the visualization (color-coding), users can rapidly identify heads that are likely to be important for specific linguistic phenomena, guiding their exploration.
  • Connecting Attention to Dependencies Reveals Syntax-Awareness: The Dependency View, when combined with attention visualizations, allows users to see how attention mechanisms align with or predict syntactic relationships, demonstrating the model's understanding of sentence structure.
  • DODRIO Facilitates Hypothesis Generation: The tool not only helps validate existing research but also inspires new hypotheses about model behavior by providing an intuitive way to explore attention patterns across different models and tasks.

Key Results

The paper presents case studies demonstrating how DODRIO can be used to:

  • Validate Coreference Resolution: In a sentiment analysis example, DODRIO helped confirm that an attention head correctly identified that the pronoun "it" referred to "film," validating a coreference resolution policy.
  • Analyze Syntactic Predictions: By visualizing attention on the Penn Treebank dataset, the authors observed patterns where specific attention heads (e.g., Layer 3, Head 9) consistently attended to nominal relationships (like object and noun modifiers), highlighting syntax-aware behavior.
  • Compare Model Architectures: A comparison between BERT and DistilBERT showed differences in attention head importance and linguistic alignment, suggesting that DistilBERT might learn linguistic knowledge beyond simple semantics and syntax in its later layers.

Limitations and Future Work

  • Subjectivity of Interpretation: The authors acknowledge the ongoing discussion about whether attention weights are true explanations or merely correlated with model behavior. DODRIO provides tools for exploration, but ultimate interpretation still requires human expertise.
  • User Study Needed: While the tool is presented with case studies, the authors plan to conduct a user study to formally evaluate its usability and usefulness.
  • Extensibility: The tool is open-sourced, encouraging users to extend it to their own models and datasets, which is a natural direction for future work.
  • Broader Impact Concerns: The authors note that while designed for good intentions, the knowledge gained from such tools could potentially be exploited by malicious actors to manipulate model predictions. Further study into this potential vulnerability is warranted.

Summary updated

Abstract

Why do large pre-trained transformer-based models perform so well across a wide variety of NLP tasks? Recent research suggests the key may lie in multi-headed attention mechanism's ability to learn and represent linguistic information. Understanding how these models represent both syntactic and semantic knowledge is vital to investigate why they succeed and fail, what they have learned, and how they can improve. We present Dodrio, an open-source interactive visualization tool to help NLP researchers and practitioners analyze attention mechanisms in transformer-based models with linguistic knowledge. Dodrio tightly integrates an overview that summarizes the roles of different attention heads, and detailed views that help users compare attention weights with the syntactic structure and semantic information in the input text. To facilitate the visual comparison of attention weights and linguistic knowledge, Dodrio applies different graph visualization techniques to represent attention weights scalable to longer input text. Case studies highlight how Dodrio provides insights into understanding the attention mechanism in transformer-based models. Dodrio is available at https://poloclub.github.io/dodrio/.
BibTeX
@inproceedings{wang2021dodrio,
  author = {Zijie J. Wang and Robert Turko and Duen Horng Chau},
  title = {{Dodrio: Exploring Transformer Models with Interactive Visualization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=HvyWvZ8VTeI},
}
Figure