Probing the Category of Verbal Aspect in Transformer Language Models

Anisia Katinskaia, Roman YangarberNAACL 2024 · DOI

TL;DR

This paper investigates how well Transformer language models (like BERT and RoBERTa) understand Russian verbal aspect, a complex grammatical feature. They found that models encode aspect primarily in their later layers and that this encoding aligns with linguistic theory, particularly concerning the concept of "boundedness."

What Problem Does This Paper Address?

Verbal aspect is a notoriously tricky grammatical category, even for human language learners. It describes how an action unfolds over time (e.g., completed vs. ongoing). In Russian, the perfective/imperfective distinction is crucial and often subtle, making it a challenging area for Natural Language Processing (NLP) models. While previous work has probed models for other linguistic features, verbal aspect in Russian has remained largely unexplored. This paper aims to bridge this gap by investigating how well Transformer models capture this complex linguistic phenomenon and whether their internal representations align with linguistic theories of aspect.

Methods

The researchers employed two main probing techniques:

  • Behavioral Probing: This involved using the models as Masked Language Models (MLMs) to predict the correct aspect form for a given context. They used two variations:
  • Iterative Masking: Gradually unmasking tokens to predict the target verb.
  • Aspect Inference: Identifying the most probable verb forms from the model's vocabulary for a masked position and checking their aspect.
  • Causal Probing: This technique involved intervening in the model's internal representations. Specifically, they manipulated the "boundedness" feature of the action described by the verb. Using a method called AlterRep, they created counterfactual representations to see how changing this feature affected the model's aspect prediction. This was done layer by layer to understand where in the model aspect information is encoded.

Datasets

The study utilized several datasets:

  • ReLCo Corpus: Used to collect "alternative contexts" where both perfective and imperfective aspects are grammatically and semantically acceptable.
  • Omnia Corpus: Used to collect sentences for creating "non-alternative contexts" and for generating data for causal probing.
  • SynTagRus Corpus: Used for fine-tuning BERT for aspect prediction.

A new dataset, the Russian Aspect Bank, was also created and released, containing over 2,000 unique aspect pairs.

Key Insights and Ideas

  • Aspect is Encoded in Later Layers: Both BERT and RoBERTa primarily encode verbal aspect in their final layers, suggesting that deeper processing is needed for this complex feature.
  • Boundedness is Key: The models' understanding of aspect is strongly linked to the concept of "boundedness" (whether an action has a limit or endpoint), aligning with linguistic theories. Manipulating this feature directly impacts aspect prediction.
  • Alternative Contexts are Tricky: Models show significantly higher uncertainty and lower performance in "alternative contexts" where multiple aspect forms are valid. This suggests these contexts lack clear cues, making it hard for models (and humans) to decide.
  • Fine-tuning Efficiency: Fine-tuning only the final layers of BERT for aspect prediction is more effective and faster than fine-tuning the entire model.
  • Causal Probing Selectivity: The causal interventions targeting boundedness did not significantly affect the prediction of other grammatical categories like number, indicating the probe's selectivity.

Key Results

  • Behavioral Probing Accuracy: BERT-large achieved up to 85-88% accuracy in predicting aspect in non-alternative contexts, primarily in its final 8 layers. Performance in alternative contexts was significantly lower.
  • Causal Probing Impact: Shifting representations towards "unbounded" improved imperfective predictions and hurt perfective ones, and vice versa for "bounded" shifts. These effects were most pronounced in the later layers. For example, negative interventions shifted imperfective accuracy by up to +21% in alternative contexts.
  • Fine-tuning Performance: Fine-tuning the last 5 layers of BERT-large for aspect prediction achieved an F0.5 score of 88.5 for perfective and 88.0 for imperfective in non-alternative contexts, comparable to its MLM performance.

Limitations and Future Work

  • Single Language Focus: The study was limited to Russian. Aspectual systems vary greatly across languages, and extending this work to other languages (like Polish) would require significant effort.
  • Dataset Size: The collected dataset, while shared, is relatively small.
  • Consensus on Aspect: There's no universal agreement among linguists on the exact nature of aspect opposition (e.g., same verb vs. different verbs).
  • Cue Word Exhaustiveness: The list of cue words for boundedness is not exhaustive, and other contextual evidence (like action completion or result observability) was not considered.
  • Future Work: The authors plan to:
  • Explore additional languages.
  • Investigate how transformers encode relationships between verbs and cue words.
  • Examine the interplay between aspect, tense, and time-related context words.
  • Study aspect prediction in contexts lacking explicit cues, requiring reasoning across sentences.
  • Deploy aspect prediction systems for language learning tools.

Summary updated

BibTeX
@inproceedings{katinskaia2024probing,
  author = {Anisia Katinskaia and Roman Yangarber},
  title = {{Probing the Category of Verbal Aspect in Transformer Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.212},
}
Figure