Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

Yan CongNAACL 2022

TL;DR

This paper investigates whether large language models (LLMs) can understand "evaluativity implicature" – the subtle, speaker-dependent meaning conveyed by gradable adjectives like "tall." By adapting experimental pragmatics methods, the study finds that while LLMs show some understanding, their performance is sensitive to adjective properties, construction type, and crucially, the persona assigned to the speaker in the prompt.

What Problem Does This Paper Address?

When humans say "Maria is tall," they usually don't just mean Maria's height exceeds a certain objective threshold. They also imply that, from their own perspective, Maria is tall in a way that carries a positive evaluation. This is known as "evaluativity implicature" (EVAL). However, in constructions like "Maria is taller than Sophie," this evaluative layer is less prominent, as it's possible for Maria to be taller than Sophie even if both are considered short. Understanding how LLMs grasp such subtle, context-dependent pragmatic inferences is a significant challenge in NLP. This paper aims to probe LLMs' ability to discern EVAL, particularly how it varies across different adjective types and linguistic constructions, and whether assigning a specific "persona" to the LLM can improve its performance.

Methods

The authors adapted a methodology from experimental pragmatics, specifically the "Bierwisch Test," to evaluate LLMs. This test posits that a construction is evaluative if it contextually entails its positive counterpart. They used this principle to measure how closely an LLM's generated response aligns with the expected evaluative inference.

Key methodological innovations include:

  • Prompt Engineering with Personas: Instead of just presenting a sentence, the LLMs were prompted with scenarios where a "speaker" (either "Nerd" or "Chill") utters a sentence with a gradable adjective. This was inspired by prior work showing that human interpretation of numerical precision is influenced by speaker persona.
  • Cosine Similarity Measurement: LLM responses were embedded into vectors, and the cosine similarity between these vectors and the target evaluative inference (the POS counterpart of the original utterance) was calculated. A higher similarity suggests the LLM captured the intended pragmatic meaning.
  • Focus on Gradable Adjectives: The study specifically targets gradable adjectives (like "tall," "short," "heavy") and their different constructions (Positive, Equative, Comparative) to dissect the nuances of EVAL.

Datasets

The study utilized a dataset derived from Brasoveanu and Rett (2018), which contains 32 gradable antonym pairs (16 relative and 16 absolute adjectives) framed in three constructions (POS, EQ, COMP), resulting in 96 unique test strings. This dataset was previously validated with human judgments on EVAL implicatures. No new datasets were introduced; instead, existing human data was used as a benchmark.

Key Insights and Ideas

  • Persona Matters: Assigning a "Chill" persona to the LLM significantly improved its ability to capture evaluativity implicatures, suggesting that pragmatic skillfulness can be "simulated" through prompt design. Conversely, a "Nerd" persona sometimes led to less nuanced interpretations.
  • Construction is Key: The study confirms human intuition that positive constructions ("Maria is tall") are the most likely to carry evaluative meaning, while equative ("as tall as") and comparative ("taller than") constructions are less so.
  • Polarity is Less Important: Similar to human judgments, LLMs did not show a significant difference in understanding EVAL between positive and negative adjectives (e.g., "tall" vs. "short").
  • Adjective Class Nuances: While humans find relative adjectives (e.g., "heavy") more evaluative in POS constructions than absolute ones (e.g., "full"), LLMs generally showed the opposite trend, considering absolute adjectives more evaluative. This is a deviation from human cognition.

Key Results

  • Alignment with Humans: LLMs generally aligned with human data regarding the higher evaluativity of POS constructions and the lack of difference based on adjective polarity.
  • Persona Impact: The "Chill" persona prompt led to improved LLM performance in capturing EVAL implicatures, especially for GPT-3.
  • Construction Sensitivity: GPT-3 demonstrated more consistent sensitivity to construction types (POS > EQ > COMP) compared to Macaw, which showed more variance, particularly with the "Nerdy" persona.
  • Adjective Class Discrepancy: Across most conditions, LLMs treated absolute adjectives as more evaluative than relative adjectives, a finding that contrasts with human judgments.

Limitations and Future Work

The authors acknowledge that using cosine similarity as a proxy for pragmatic inference might not be fine-grained enough to fully capture subtle differences. Future work could involve using Natural Language Inference (NLI) models for more direct testing of contextual entailment. They also suggest expanding the dataset and further exploring the sociolinguistic persona design to better understand LLM behavior. The study highlights the value of leveraging existing human behavioral data to evaluate LLMs' strengths and weaknesses.

Summary updated

BibTeX
@inproceedings{cong2022pre,
  author = {Yan Cong},
  title = {{Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure