PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities

Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, Raj Dabre

TL;DR

While Large Language Models (LLMs) excel at capturing the literal meaning of text (semantics), they frequently struggle with the nuances of context, intent, and implied meaning (pragmatics). This paper introduces the Pragmatics Understanding Benchmark (PUB), a 28k-example dataset designed to evaluate LLMs on real-world conversational phenomena like sarcasm, presupposition, and indirect answers. The findings reveal a significant performance gap between humans and AI, showing that models often rely on superficial linguistic cues rather than true pragmatic reasoning.

What Problem Does This Paper Address?

Modern LLMs achieve remarkable scores on standard semantic and problem-solving benchmarks, but human conversation is rarely entirely literal. If someone asks, "Have you ever been to a live rock gig?" and the response is, "I went to a Taylor Swift concert once," a human easily infers an indirect "yes." This is the domain of pragmatics—how context shapes meaning.

Currently, there is a lack of comprehensive, linguistically grounded benchmarks to test whether LLMs actually understand speaker intentions, implied meanings (implicatures), presuppositions, and context-dependent references (deixis). The authors address this gap by creating a unified evaluation suite to determine if LLMs can bridge the gap between what is said and what is actually meant.

Methods

The authors evaluate nine open-source and proprietary models (including Llama-2 variants, T5, Flan-T5, and GPT-3.5) using two prompting strategies:

  1. Multiple Choice Prompting (MCP): Standard multiple-choice question answering.
  2. Cloze Prompting (CP): Length-normalized fill-in-the-blank style prompting.

To ensure models aren't just exploiting the order of multiple-choice options, the authors use a metric called the Proportion of Plurality Agreement (PPA) to validate prediction consistency across shuffled choices. Finally, they establish a human baseline by having fluent English-speaking graduate students evaluate a randomized subset of 100 examples per task.

Datasets

PUB aggregates 28,000 data points across 14 distinct tasks. While it adapts several existing datasets (such as GRICE, FigQA, and FLUTE), it introduces three entirely new, expert-annotated datasets totaling 6,100 examples to fill critical gaps:

  1. CircaPlus: 2.5k human-written implied meanings derived from indirect conversational responses.
  2. DialogAssumptions: 2.5k conversational presupposition pairs where explicit trigger words are completely absent.
  3. MetoQA: 1,100 multiple-choice questions testing metonymy (a figure of speech where an concept is replaced by a closely related term, e.g., using "the robes" to refer to "judges").

An overview of how these tasks are structured across the four pragmatic domains is illustrated below:

Figure 2: Examples of each task from PUB, The tasks are divided across four domains of pragmatics (Implicature, Presupposition, Reference, and Deixis). Our proposed benchmark builds upon existing pragmatic datasets and combines our newly annotated datasets comprising 6k annotations to complete the pragmatic evaluation test suite with 28k examples. We have reformatted the existing datasets into MCQA prompts that explicitly test these abilities.

Key Insights and Ideas

  • Linguistic Cues over Logic: LLMs are highly sensitive to superficial prompt variations. Providing a "positive hint" boosts performance, but introducing a "contrastive hint" causes a drastic accuracy drop (~20% on average). This suggests models rely heavily on surface-level semantic overlap rather than robust logical reasoning.
  • The Scaling Paradox for Chat Models: Instruction and chat fine-tuning significantly enhance the pragmatic capabilities of smaller models (e.g., 7B parameters). However, for larger models like Llama-2-70B, the base versions perform comparably to their chat-adapted counterparts.
  • It's Not a Knowledge Bottleneck: The primary issue isn't a lack of world knowledge. Models perform poorly even on tasks that do not rely on world knowledge (like Deixis/pointing words), indicating a fundamental deficiency in innate pragmatic processing.
  • Systematic Pessimism: Error analysis reveals that LLMs have a strong tendency to misclassify positive pragmatic labels as negative, frequently failing to recognize tactful "polite declines" or complex figurative agreements.

Key Results

  • The Human-AI Gap: A stark performance gap remains between humans and LLMs across all pragmatic categories. While human evaluators consistently score near or above 80–90% accuracy, model performance fluctuates wildly depending on the task complexity.
Average performance of models vs. human evaluators across different pragmatics phenomena.
  • Sarcasm vs. Agreement: Humans achieve 95–97% accuracy on sarcasm and agreement detection. In contrast, LLMs show a massive performance discrepancy (a ~13% gap for models $\ge$ 13B parameters), proving much better at recognizing straightforward agreement than detecting sarcasm within the same dataset.
  • Flan-T5 Punching Above Its Weight: The instruction-tuned flan-t5-xxl model consistently approaches near-human-level performance on several tasks, frequently outperforming larger base models.

Limitations and Future Work

The authors note that LLMs remain highly sensitive to exact prompt wordings and option ordering, which introduces variance into reproducibility. Furthermore, the human baseline was established using a specific demographic (engineering graduate students), meaning human benchmarks could vary across broader populations.

The authors suggest that future research should focus deeply on why LLMs fail so easily in trivial pragmatic cases that humans solve effortlessly, paving the way for more naturally interactive conversational agents.

Summary updated

Abstract

LLMs have demonstrated remarkable capability for understanding semantics, but they often struggle with understanding pragmatics. To demonstrate this fact, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely, Implicature, Presupposition, Reference, and Deixis. We curated high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA). PUB includes a total of 28k data points, 6.1k of which have been created by us, and the rest are adapted from existing datasets. We evaluated nine models varying in the number of parameters and type of training. Our study indicates that fine-tuning for instruction-following and chat significantly enhances the pragmatics capabilities of smaller language models. However, for larger models, the base versions perform comparably with their chat-adapted counterparts. Additionally, there is a noticeable performance gap between human capabilities and model capabilities. Furthermore, unlike the consistent performance of humans across various tasks, the models demonstrate variability in their proficiency, with performance levels fluctuating due to different hints and the complexities of tasks within the same dataset. Overall, the benchmark aims to provide a comprehensive evaluation of LLM's ability to handle real-world language tasks that require pragmatic reasoning.
BibTeX
@inproceedings{sravanthi2024pub,
  author = {Settaluri Lakshmi Sravanthi and Meet Doshi and Tankala Pavan Kalyan and Rudra Murthy and Pushpak Bhattacharyya and Raj Dabre},
  title = {{PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.719},
}
Figure