Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity

Yuxi Li, Emmanuele Chersoni, Yu-Yin HsuNAACL 2024

TL;DR

This paper investigates how well modern language models like BERT and GPT-2 capture the nuances of verb aspect (stativity, telicity, durativity). They found these models are surprisingly good at stativity and telicity, but struggle with durativity. Interestingly, only BERT's early layers could replicate a classic linguistic paradox related to verb aspect, suggesting a deeper understanding of temporal semantics.

What Problem Does This Paper Address?

Understanding how language models represent linguistic knowledge is a crucial area of research. One fundamental aspect of language is verb aspect, which describes how actions and events unfold over time. Traditionally, aspect is characterized by three properties: stativity (states vs. events), telicity (whether an event has a natural endpoint), and durativity (how long an event lasts). While modern contextualized embeddings (like those from BERT and GPT-2) have shown promise in capturing various linguistic features, it's unclear to what extent they encode these subtle aspectual distinctions. This paper aims to quantify this knowledge and explore how it's distributed across different layers of these models.

Methods

The researchers employed two main experimental approaches:

  1. Semantic Scales and Projection: They adapted a method from Grand et al. (2022) to create "semantic scales" for stativity, telicity, and durativity. These scales were derived by averaging word embeddings of verbs known to represent extreme values for each property (e.g., "exist" vs. "dance" for stativity). Then, they projected the embeddings of other verbs onto these scales to assign them scores for each aspectual property. The hypothesis was that if the models encoded these properties, verbs belonging to different categories (e.g., telic vs. atelic) would receive significantly different scores.
  2. Distributional Similarity and the Imperfective Paradox: To test the models' understanding of temporal relationships and entailment, they investigated the "Imperfective Paradox" (Dowty, 1977). This paradox states that for activity verbs (atelic), the past progressive form entails the simple past (e.g., "Maria was singing" implies "Maria sang"), but this entailment doesn't hold for accomplishment verbs (telic) (e.g., "The children were building a sandcastle" does not necessarily imply "The children built a sandcastle"). They measured the distributional similarity between simple past and past progressive verb embeddings for telic and atelic verbs. They hypothesized that atelic verbs would show higher similarity, reflecting the entailment relationship.

Datasets

The study utilized verbs categorized according to Vendler's (1957) four classes: states, activities, accomplishments, and achievements. For generating contextualized embeddings, they used sentences from the British National Corpus (BNC). The specific verbs used for scale construction and projection were manually verified after being generated by ChatGPT.

Key Insights and Ideas

  • Models are surprisingly good at stativity and telicity: Both BERT and GPT-2 embeddings show a strong ability to distinguish between stative/dynamic verbs and telic/atelic verbs, with this knowledge present across most layers.
  • Durativity is the Achilles' heel: Durativity proved to be the most challenging aspectual property for both models to consistently encode.
  • BERT's early layers capture the Imperfective Paradox: Only BERT's early layers successfully replicated the expected pattern for the Imperfective Paradox, suggesting that deeper layers might become too context-dependent to reflect this specific entailment.
  • Anisotropy's impact: The study highlights how anisotropy (the tendency for embeddings to cluster in a small cone of space) can affect similarity measures, particularly in later layers of autoregressive models like GPT-2, making them less reliable for tasks requiring fine-grained semantic distinctions.
  • Layer-wise knowledge distribution: The research provides a detailed view of how aspectual knowledge is distributed across the layers of BERT and GPT-2, showing that while some knowledge is present early on, it can shift or become less clear in deeper layers.

Key Results

  • Experiment 1 (Semantic Scales):
    • Stativity: Differences between stative and dynamic verbs were significant across almost all layers for both BERT and GPT-2.
  • Telicity: Differences were significant for most BERT layers. For GPT-2, telicity became more salient in deeper layers but dropped to non-significance in the last layer.
  • Durativity: This was the most challenging property. BERT showed significant differences in some early, middle, and later layers, while GPT-2 only distinguished the groups in early and middle layers, failing in the later ones.
  • Experiment 2 (Imperfective Paradox):
    • BERT: Cosine similarity between past and past progressive forms of atelic verbs was generally higher than for telic verbs, especially in earlier layers, aligning with the hypothesis.
  • GPT-2: Showed the opposite pattern, with telic verbs having higher similarity than atelic verbs across most layers. This behavior, along with scores approaching 1 in later layers, is attributed to anisotropy.

Limitations and Future Work

  • English-centric: The study was conducted solely on English. Future work could explore whether these findings generalize to other languages and Transformer models.
  • Binary Distinctions: The experiments assumed binary distinctions for aspectual properties and entailment. The authors acknowledge that human intuitions can be more nuanced, and future studies might benefit from a "perspectivist approach" to account for disagreements in human judgments.
  • Limited Model Scope: Only BERT and GPT-2 were tested. The authors suggest testing more recent Large Language Models (LLMs).
  • Future Directions: The authors suggest exploring how to account for differences in human semantic intuitions and investigating the impact of context on aspectual representations more deeply. The qualitative analysis of outliers in Experiment 2 also points to the complex interplay of telicity, durativity, and context in determining verb behavior.

Summary updated

BibTeX
@inproceedings{li2024investigating,
  author = {Yuxi Li and Emmanuele Chersoni and Yu-Yin Hsu},
  title = {{Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure