TL;DR
While Large Language Models (LLMs) excel at capturing the literal meaning of text (semantics), they frequently struggle with the nuances of context, intent, and implied meaning (pragmatics). This paper introduces the Pragmatics Understanding Benchmark (PUB), a 28k-example dataset designed to evaluate LLMs on real-world conversational phenomena like sarcasm, presupposition, and indirect answers. The findings reveal a significant performance gap between humans and AI, showing that models often rely on superficial linguistic cues rather than true pragmatic reasoning.
What Problem Does This Paper Address?
Modern LLMs achieve remarkable scores on standard semantic and problem-solving benchmarks, but human conversation is rarely entirely literal. If someone asks, "Have you ever been to a live rock gig?" and the response is, "I went to a Taylor Swift concert once," a human easily infers an indirect "yes." This is the domain of pragmatics—how context shapes meaning.
Currently, there is a lack of comprehensive, linguistically grounded benchmarks to test whether LLMs actually understand speaker intentions, implied meanings (implicatures), presuppositions, and context-dependent references (deixis). The authors address this gap by creating a unified evaluation suite to determine if LLMs can bridge the gap between what is said and what is actually meant.
Methods
The authors evaluate nine open-source and proprietary models (including Llama-2 variants, T5, Flan-T5, and GPT-3.5) using two prompting strategies:
- Multiple Choice Prompting (MCP): Standard multiple-choice question answering.
- Cloze Prompting (CP): Length-normalized fill-in-the-blank style prompting.
To ensure models aren't just exploiting the order of multiple-choice options, the authors use a metric called the Proportion of Plurality Agreement (PPA) to validate prediction consistency across shuffled choices. Finally, they establish a human baseline by having fluent English-speaking graduate students evaluate a randomized subset of 100 examples per task.
Datasets
PUB aggregates 28,000 data points across 14 distinct tasks. While it adapts several existing datasets (such as GRICE, FigQA, and FLUTE), it introduces three entirely new, expert-annotated datasets totaling 6,100 examples to fill critical gaps:
- CircaPlus: 2.5k human-written implied meanings derived from indirect conversational responses.
- DialogAssumptions: 2.5k conversational presupposition pairs where explicit trigger words are completely absent.
- MetoQA: 1,100 multiple-choice questions testing metonymy (a figure of speech where an concept is replaced by a closely related term, e.g., using "the robes" to refer to "judges").
An overview of how these tasks are structured across the four pragmatic domains is illustrated below:
Key Insights and Ideas
- Linguistic Cues over Logic: LLMs are highly sensitive to superficial prompt variations. Providing a "positive hint" boosts performance, but introducing a "contrastive hint" causes a drastic accuracy drop (~20% on average). This suggests models rely heavily on surface-level semantic overlap rather than robust logical reasoning.
- The Scaling Paradox for Chat Models: Instruction and chat fine-tuning significantly enhance the pragmatic capabilities of smaller models (e.g., 7B parameters). However, for larger models like Llama-2-70B, the base versions perform comparably to their chat-adapted counterparts.
- It's Not a Knowledge Bottleneck: The primary issue isn't a lack of world knowledge. Models perform poorly even on tasks that do not rely on world knowledge (like Deixis/pointing words), indicating a fundamental deficiency in innate pragmatic processing.
- Systematic Pessimism: Error analysis reveals that LLMs have a strong tendency to misclassify positive pragmatic labels as negative, frequently failing to recognize tactful "polite declines" or complex figurative agreements.
Key Results
- The Human-AI Gap: A stark performance gap remains between humans and LLMs across all pragmatic categories. While human evaluators consistently score near or above 80–90% accuracy, model performance fluctuates wildly depending on the task complexity.
- Sarcasm vs. Agreement: Humans achieve 95–97% accuracy on sarcasm and agreement detection. In contrast, LLMs show a massive performance discrepancy (a ~13% gap for models $\ge$ 13B parameters), proving much better at recognizing straightforward agreement than detecting sarcasm within the same dataset.
- Flan-T5 Punching Above Its Weight: The instruction-tuned
flan-t5-xxlmodel consistently approaches near-human-level performance on several tasks, frequently outperforming larger base models.
Limitations and Future Work
The authors note that LLMs remain highly sensitive to exact prompt wordings and option ordering, which introduces variance into reproducibility. Furthermore, the human baseline was established using a specific demographic (engineering graduate students), meaning human benchmarks could vary across broader populations.
The authors suggest that future research should focus deeply on why LLMs fail so easily in trivial pragmatic cases that humans solve effortlessly, paving the way for more naturally interactive conversational agents.
Summary updated