Exploring Semantics in Pretrained Language Model Attention

Frédéric Charpentier, Jairo Cugliari, Adrien GuilleNAACL 2024

TL;DR

This paper investigates whether the attention mechanisms within large language models (LLMs) like RoBERTa and GPT-2 can inherently capture semantic relationships without explicit fine-tuning. The authors demonstrate that attention heads show a surprising correlation with these semantic roles, with bidirectional models like RoBERTa performing notably better than unidirectional ones like GPT-2.

What Problem Does This Paper Address?

Abstract Meaning Representations (AMRs) are graph-based structures that capture the meaning of sentences, abstracting away from syntax. While LLMs have shown remarkable performance on various NLP tasks, understanding how they process and represent meaning, particularly through their attention mechanisms, remains an active area of research. This paper aims to bridge this gap by exploring whether the attention heads of pre-trained LLMs can directly identify semantic relationships encoded in AMR graphs, without requiring task-specific fine-tuning. This is crucial for developing more interpretable and trustworthy AI systems.

Methods

The core of the methodology involves using pre-trained LLMs (RoBERTa and GPT-2) and analyzing their attention mechanisms. The authors created a dataset of token pairs from sentences, aligned with vertices (concepts) in AMR graphs, and the semantic relations (edges) between them. They then used two main approaches:

  • Unsupervised Analysis (PCA): Principal Component Analysis (PCA) was applied to the attention features to reduce dimensionality and visualize the distributions of different semantic relations. This helped identify which relations were more easily distinguishable by the attention mechanism.
  • Supervised Analysis (Logistic Classifier): A logistic regression classifier was trained to predict the semantic relation between token pairs based on their attention features. This allowed for a more quantitative evaluation of the LLMs' ability to detect these relations.

They specifically focused on the "inner products" of attention weights before the softmax normalization, as these are thought to represent the raw attention scores.

Datasets

The primary dataset used was derived from the LDC2020T02 dataset, which contains 59,255 sentences paired with their corresponding AMR graphs. The authors leveraged existing alignments between AMR graphs and English sentences published by Blodgett and Schneider (2021). They filtered these alignments to create a dataset specifically for evaluating semantic relation detection, including pairs with and without a semantic relation (<no_rel>).

Key Insights and Ideas

  • Attention Heads Encode Semantics: The study reveals a significant correlation between LLM attention heads and the detection of semantic relations in AMR graphs, even without task-specific fine-tuning.
  • Bidirectional vs. Unidirectional: RoBERTa, a bidirectional model, consistently outperformed GPT-2, a unidirectional decoder, in identifying semantic relations. This suggests that the ability to attend to both past and future tokens is crucial for capturing deeper semantic nuances.
  • PCA Reveals Distinguishable Relations: Unsupervised analysis using PCA showed that certain semantic relations (like ARGO, ARG1, time, condition, location) have distinct distributions in the attention space, indicating they are more readily identifiable.
  • Specialized Attention Heads: The analysis of logistic classifier coefficients suggests that specific attention heads are specialized for either identifying a semantic relation or detecting its absence (<no_rel>).
  • Layer and Head Importance: The most discriminative attention heads tend to be located in the middle layers of the transformer, with a notable imbalance in favor of the "source-to-target" (ST) attention direction for distinguishing relations.

Key Results

  • Supervised Classification: A logistic classifier trained on RoBERTa's attention features achieved a global balanced accuracy of 0.62. Individual F1 scores for relations like ARGO and ARG1 were as high as 0.74 and 0.63, respectively.
  • GPT-2 Performance: GPT-2, using only unidirectional attention, achieved a global balanced accuracy of 0.44. Augmenting GPT-2 to use all inner products (before masking) improved its performance to 0.56, highlighting the importance of full attention information.
  • PCA Visualization: Figure 3 clearly shows that RoBERTa's attention distributions for relations like <no_rel>, ARGO, and ARG1 are much more separated than those of GPT-2, visually confirming RoBERTa's superior ability.

Limitations and Future Work

  • Model Choice: The study focused on RoBERTa and GPT-2. Further research could explore other LLMs, including more recent and larger models.
  • Linear Classifier: While a logistic classifier allows for interpretability, more complex models might achieve higher accuracy. The authors acknowledge that their linear methods reveal an "important fact" about semantic encoding, but future work could explore non-linear approaches.
  • Dataset Size and Balance: The dataset, while substantial, has imbalanced class distributions for certain relations, which was addressed by grouping less frequent relations into <other>. Future work could involve creating more balanced datasets.
  • Interpretation of Heads: While the study identified specialized heads, a deeper understanding of what specific semantic features each head captures remains an open question. The authors suggest that the "linear fashion" in which LMs encode semantic structure could be further exploited.

Summary updated

BibTeX
@inproceedings{charpentier2024exploring,
  author = {Frédéric Charpentier and Jairo Cugliari and Adrien Guille},
  title = {{Exploring Semantics in Pretrained Language Model Attention}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure