Evaluating Lexical Aspect with Large Language Models

Bolei MaACL 2024

TL;DR

This paper investigates how well large language models (LLMs) can understand the nuances of lexical aspect in English verbs, specifically focusing on duration (durative/stative) and telicity (telic/atelic). The study finds that while LLMs show promise, they generally lag behind fine-tuned traditional models, though providing more contextual information about the verb can improve their performance.

What Problem Does This Paper Address?

Understanding the temporal nature of events and states described by verbs is crucial for a deep comprehension of language. This is where "lexical aspect" comes in. It's not about when an action happens (tense), but how it unfolds in time. For instance, "running" is a durative action (ongoing), while "knowing" is a stative aspect (a state). Similarly, "eating an apple" is telic (it has an endpoint), whereas "eating" in general is atelic (no specific endpoint).

While linguists have studied these concepts for decades, computationally identifying them has been challenging. Previous work relied on specialized NLP models trained on specific datasets. With the rise of powerful LLMs, this paper asks: can these general-purpose models, without explicit fine-tuning for aspect, grasp these subtle linguistic features? The challenge is to see if LLMs can move beyond surface-level understanding to capture these deeper semantic properties of verbs within sentences.

Methods

The researchers employed a zero-shot prompting approach to evaluate six different LLMs (GPT-3.5, GPT-4, Llama-2, Llama-3, Gemma, and Mixtral). They designed three distinct prompting levels to test the models' understanding:

  1. Sentence-only: The LLM receives only the sentence and is asked to identify the aspect.
  2. Sentence with verb: The LLM receives the sentence and the specific verb in question.
  3. Sentence with verb and its position: The LLM receives the sentence, the verb, and its exact position within the sentence.

This tiered approach aimed to assess how much contextual information about the verb itself influences the LLM's ability to correctly identify aspectual features. Outputs from the LLMs were then processed using string matching to categorize their predictions.

Datasets

The study utilized a dataset previously created by Metheniti et al. (2022), which itself was built upon datasets from Friedrich and Gateva (2017) and Alikhani and Stone (2019). This dataset contains sentences annotated for two main aspectual features:

  • Duration: Binary labels for durative (action) or stative (state).
  • Telicity: Binary labels for telic (has an endpoint) or atelic (no endpoint).

The paper does not introduce a new dataset but rather leverages existing, well-established resources for aspectual classification.

Key Insights and Ideas

  • LLMs show potential but aren't perfect: While LLMs can recognize lexical aspect, their performance is generally lower than traditional, fine-tuned models. This suggests that while LLMs have broad linguistic knowledge, they might need specific adaptation for nuanced tasks like aspectual classification.
  • Context is king (mostly): Providing more information about the verb (its identity and position) generally improves LLM performance. This indicates that LLMs benefit from explicit cues when dealing with subtle semantic features.
  • Telicity is trickier than duration: Models performed slightly worse on telicity tasks compared to duration tasks, suggesting that identifying endpoints is a more challenging problem for LLMs.
  • Closed-source models lead the pack: GPT-3.5 and GPT-4 consistently outperformed their open-source counterparts, highlighting a current advantage in proprietary LLM development for this task.
  • Prompting can be a double-edged sword: While more context helps, it can also introduce instability and even lead to worse performance in some cases, suggesting that LLMs can sometimes be sensitive to prompt complexity.

Key Results

The study found that:

  • GPT-4 achieved the highest accuracy (0.74) and F1 score (0.76) on the duration test set among the LLMs.
  • On the telicity test set, GPT-4 again led with an accuracy of 0.71 and an F1 score of 0.72.
  • The fine-tuned bert-large-cased model served as a strong baseline, significantly outperforming all LLMs with accuracies of 0.96 (duration) and 0.88 (telicity).
  • Including verb information (Level 2) and verb position (Level 3) generally improved F1 scores across models compared to sentence-only prompts (Level 1). For example, GPT-4 showed substantial improvements in corrections over regressions when additional context was provided.
  • However, some models, like Llama-2 on the telicity set, showed more regressions than corrections with added context, indicating potential confusion.

Limitations and Future Work

The authors acknowledge several limitations:

  • LLM Instability: LLMs can be sensitive to prompt wording and parameter settings, leading to variations in performance.
  • Limited Dataset Scope: The study used a single dataset with individual sentences. Future work could explore longer texts and sequential predictions to better understand how LLMs handle aspect within broader contexts.
  • Zero-Shot Limitations: The current study relied solely on zero-shot prompting. Previous research suggests this might underestimate LLMs' capabilities.

Future research directions include:

  • Fine-tuning LLMs: Adapting LLMs specifically for aspectual feature recognition.
  • Few-shot prompting and Chain-of-Thought: Exploring more advanced prompting techniques to elicit deeper reasoning.
  • Incorporating Linguistic Knowledge: Integrating explicit linguistic information into LLM training.
  • Investigating Prompt Sensitivity: Delving deeper into why certain prompts lead to better or worse results.

Summary updated

BibTeX
@inproceedings{ma2024evaluating,
  author = {Bolei Ma},
  title = {{Evaluating Lexical Aspect with Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure