Modeling the Influence of Verb Aspect on the Activation of Typical Event Locations with BERT

Won Ik Cho, Emmanuele Chersoni, Yu-Yin Hsu, Chu-Ren HuangACL 2021

TL;DR

This paper investigates whether BERT, a powerful language model, understands how verb aspect (e.g., ongoing vs. completed actions) influences our understanding of typical event locations. While BERT accurately predicts typical locations, it does so independently of verb aspect in a standard setting. However, when forced to focus solely on the verb phrase, BERT shows some sensitivity to aspect, associating ongoing actions with less surprising locations.

What Problem Does This Paper Address?

Understanding natural language involves more than just recognizing words; it requires activating generalized event knowledge. For instance, the verb "open" might trigger expectations of an "door" as an argument. Previous research has shown that verb aspect plays a crucial role in how we process non-core semantic roles, such as locations. Specifically, imperfective aspect (describing an ongoing event) makes locations more salient and easier to process compared to perfective aspect (describing a completed event). This study aims to determine if BERT, a state-of-the-art language model, exhibits similar aspect-driven behavior in predicting typical event locations.

Methods

The researchers used BERT (specifically, the bert-base-uncased model) to model two datasets derived from prior experimental studies. They employed a "surprisal score" as a measure of BERT's expectations for a given location. This score is calculated by masking the location word in a sentence and observing the probability BERT assigns to it.

Two experimental settings were used:

  1. Standard Setting: BERT had access to the entire sentence context to predict the masked location.
  2. Context Mask Setting: BERT's attention mechanism was restricted to only the verb phrase, forcing it to focus on the verb's immediate context to predict the location. This was done to isolate the effect of verb aspect without interference from other sentence elements.

Datasets

The study utilized two datasets:

  1. Ferretti07: Derived from an EEG experiment by Ferretti et al. (2007), this dataset consists of 32 verb-location pairs where the verb phrase was specifically biased towards a locative prepositional phrase. Items were categorized by verb aspect (perfective/imperfective) and location typicality (typical/non-typical).
  2. Ferretti01: Created by the authors based on typicality judgments from Ferretti et al. (2001). This dataset comprises 135 verb-argument pairs, with sentences constructed to vary in verb aspect (perfective/imperfective) but focusing on typical locations.

Key Insights and Ideas

  • BERT's Location Typicality Prowess: BERT is remarkably good at identifying typical event locations, even when differentiating them from plausible but less common ones. This ability appears to be independent of verb aspect in a standard sentence context.
  • Aspect's Subtlety: Unlike human processing where imperfective aspect clearly enhances location salience, BERT's standard processing doesn't show a strong aspect-driven effect on location typicality. This suggests that BERT might be leveraging other semantic cues more heavily.
  • Forced Focus Reveals Aspect: When BERT is constrained to focus only on the verb phrase (context mask setting), verb aspect does start to play a role. Imperfective aspect is associated with lower surprisal scores for locations, implying that ongoing actions lead to less surprising location predictions for BERT in this constrained scenario.
  • No Interaction with Typicality: Crucially, BERT did not replicate the human finding where aspect-specific expectations for locations only emerge with imperfective verbs. This indicates a difference in how BERT and humans integrate aspectual information with typicality.
  • Word Frequency's Limited Role: In the standard setting, word frequency of the location noun had little to no impact on BERT's predictions. However, in the context mask setting, a weak inverse correlation emerged, suggesting frequency might play a minor role when contextual cues are reduced.

Key Results

  • Standard Setting: No significant main effect of verb aspect was found on location surprisal scores in either dataset. BERT accurately identified typical locations, with typical locations consistently showing lower surprisal than non-typical ones (p < 0.001 in Ferretti07).
  • Context Mask Setting:
    • In Ferretti01, a marginally significant effect of aspect was observed, with lower surprisal for imperfective conditions (p < 0.1).
  • In Ferretti07, significant main effects for both typicality (p < 0.01) and aspect (p < 0.05) were found. Imperfective aspect was associated with significantly lower surprisal scores for locations (p < 0.05).
  • Qualitative Finding: In a few instances (Example 4), BERT assigned lower surprisal to non-typical but more generic/frequent fillers over specific, typical ones, mirroring findings in other NLP tasks.

Limitations and Future Work

  • Human-vs-BERT Differences: The study highlights that BERT's aspectual processing, particularly its interaction with typicality, differs from human behavior. BERT did not replicate the specific interaction found in Ferretti et al. (2007) where aspect-specific expectations for locations were prominent only in imperfective sentences.
  • Subtle Nuances: While BERT's predictions were somewhat sensitive to aspect-related differences in the context mask setting, the authors suggest these might reflect "subtle nuances of argument typicality."
  • Future Directions: The authors hope their work opens new avenues for investigating the interplay between grammar (like verb aspect) and event knowledge using modern language models. Further research could explore other grammatical features and semantic roles to better understand how these models capture linguistic phenomena.

Summary updated

BibTeX
@inproceedings{cho2021modeling,
  author = {Won Ik Cho and Emmanuele Chersoni and Yu-Yin Hsu and Chu-Ren Huang},
  title = {{Modeling the Influence of Verb Aspect on the Activation of Typical Event Locations with BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure