Adverbs, Surprisingly

Dmitry Nikolaev, Collin Baker, Miriam R. L. Petruck, Sebastian PadóACL 2023

TL;DR

This paper argues that adverbs are surprisingly under-addressed in computational linguistics, leading to systematic gaps in current NLP models. The authors propose using Frame Semantics, as implemented in FrameNet, as a promising framework for capturing adverbial meaning, and demonstrate the limitations of current large language models (LLMs) on this task through a novel probing dataset.

What Problem Does This Paper Address?

Despite their crucial role in conveying nuanced meaning, adverbs have historically received limited attention in computational linguistics. This neglect stems from their inherent complexity: they modify various parts of speech, exhibit semantic variety (from intensifiers to speaker evaluations), and can occupy multiple positions in a sentence, leading to scoping and ambiguity issues. Existing NLP resources like WordNet offer impoverished accounts, and specialized lexicons often focus on narrow adverb subtypes. Consequently, state-of-the-art LLMs, while powerful, struggle to accurately interpret adverbial meaning, as evidenced by their performance on natural language inference tasks. This paper aims to bridge this gap by proposing a more robust framework for adverb analysis and empirically demonstrating the shortcomings of current models.

Methods

The authors propose leveraging Frame Semantics, specifically the FrameNet resource, as a principled approach to understanding adverb meaning. FrameNet's core concept is the semantic frame, a knowledge structure that captures the meaning of a lexical unit (LU) and its associated frame elements (FEs), which represent semantic roles. The paper argues that FrameNet's ability to describe ambiguity (through different frames for different senses of an adverb), semantic roles, and null instantiation (unexpressed but implied FEs) makes it well-suited for adverbs.

To empirically evaluate current LLMs, the authors created probing datasets for natural language inference (NLI). These datasets were designed to test how well LLMs can infer meaning from sentences containing speaker-oriented adverbs. They constructed templates for four classes of adverbs: likelihood, unattributed-information, degree, and obviousness. These templates were populated with various fillers to assess the LLMs' understanding irrespective of specific word choices. The evaluation focused on whether LLMs could correctly distinguish between entailment, contradiction, and neutral relationships between premise and hypothesis sentences containing these adverbs.

Datasets

The paper introduces two novel datasets for evaluating LLMs on adverb understanding:

  • Naturalistic Dataset: This dataset was created by "templatizing" existing sentences from the FrameNet database that contained adverbs from the four target classes.
  • Synthetic Dataset: A larger, more varied dataset was generated synthetically to test the robustness of LLM performance across different sentence structures and adverb classes. This dataset focused on unattributed-information, degree, and obviousness adverbs.

These datasets were used to probe existing Natural Language Inference (NLI) models, specifically the ROBERTa-large model fine-tuned on the SNLI dataset. The authors emphasize that their datasets were used solely for evaluation, not for further fine-tuning.

Key Insights and Ideas

  • Adverbs are surprisingly neglected: Despite their semantic richness, adverbs are a "residual category" in linguistics and have not received adequate attention in computational linguistics.
  • Frame Semantics as a solution: FrameNet's framework, with its emphasis on semantic frames, frame elements, and null instantiation, offers a powerful way to capture the nuances of adverbial meaning, including ambiguity and implied information.
  • LLMs struggle with adverbial inference: Current LLMs, even powerful ones like RoBERTa-large, exhibit significant systematic shortcomings in drawing correct inferences from sentences containing adverbs, particularly speaker-oriented ones.
  • Structural complexity exacerbates adverbial challenges: The presence of adverbs, especially when combined with embedded clauses in hypotheses, can significantly degrade LLM performance on NLI tasks.
  • Adverbial meaning is often overlooked: In some cases, LLMs appear to ignore adverbs altogether, making predictions based primarily on the main participants of a sentence, suggesting a bias in their training data.
  • Intentional data creation is crucial: The authors argue that simply increasing the size of general-purpose training datasets is unlikely to resolve the adverb understanding problem; specialized, intentionally created datasets are necessary.

Key Results

  • The evaluation of the RoBERTa-large model on the naturalistic dataset revealed that LLMs fail to correctly draw inferences based on certain adverbs and that the presence of adverbs generally increases the difficulty for the model to make correct inferences.
  • Specific adverb classes showed higher error rates, with degree and obviousness adverbs being particularly challenging for the model.
  • The model often made predictions with high confidence even when incorrect, indicating a lack of true understanding rather than simple uncertainty.
  • In some instances, the model's performance was worse with adverbs than without them, suggesting that adverbs can actively mislead the model.
  • The synthetic dataset further confirmed that LLMs tend to predict the same relation (entailment/neutral/contradiction) for all sentences within an adverb class, indicating a failure to capture nuanced adverbial meaning.

Limitations and Future Work

  • Language Limitation: The study was conducted solely on English data, and the authors acknowledge that findings may not generalize to other languages, although they believe the core challenges are comparable.
  • Dataset Size: While efforts were made to create comprehensive datasets, the limited number of sentences in the naturalistic dataset and the specific focus of the synthetic dataset are noted.
  • Future Work: The authors propose extending FrameNet with more adverbs and frames, developing theoretical frameworks to precisely account for null-instantiated adverbial FEs, and creating more data. They suggest that this work can inform the development of better semantic role labelers and the fine-tuning of LLMs on large, specialized adverb datasets. They also highlight the potential of crowdsourcing and automatic methods for annotation to mitigate the resource-intensive nature of this task.

Summary updated

BibTeX
@inproceedings{nikolaev2023adverbs,
  author = {Dmitry Nikolaev and Collin Baker and Miriam R. L. Petruck and Sebastian Padó},
  title = {{Adverbs, Surprisingly}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure