The Challenge of Pragmatic Reasoning in Language Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2142 papers192 conceptsfrozen 2026-10-04

Large language models (LLMs) have become remarkably proficient at the "what" of language—grammar, syntax, and literal dictionary definitions. However, they frequently struggle with the "how" and "why"—the field of pragmatics. Pragmatics is the study of how context, social norms, and speaker intent transform the literal meaning of words into a specific communicative goal.

While a model might correctly define the word "chilly," it often fails to realize that a user saying "It's chilly in here" is actually making an indirect request to turn up the heat. This "pragmatic gap" represents a fundamental hurdle in moving from statistical text predictors to true conversational partners.

The Divergence of Literal and Intended Meaning

The core difficulty lies in the distance between literal semantics (the surface meaning) and pragmatic interpretation (the subtext). As Yuxin Wang and colleagues demonstrate with the ImpScore metric, as language becomes more implicit—relying on sarcasm, hints, or coded language—model performance in intent detection and safety tasks drops significantly.

This failure is not just a matter of "knowing" more words. Bolei Ma notes that even as models scale in size, they continue to struggle with reading between the lines in high-stakes social interactions. The models treat language as a literal translation of data rather than a tool for achieving a social goal.

Scalar and Conversational Implicature

One of the most common pragmatic hurdles is implicature: meaning that is suggested rather than stated.

  1. Scalar Implicature: If a speaker says a movie was "good," humans pragmatically infer it was "not great." Rashid Nizamani and Sebastian Schuster found that models like DeBERTa often mistake these subtle distinctions for simple synonyms, failing to recognize that choosing a weaker term often negates a stronger one.
  2. Conversational Implicature: This involves following (or intentionally flouting) the Gricean Maxims—rules like "be relevant" and "be informative." Laura Ruis and Tim Rocktäschel show that most LLMs perform no better than a coin flip at resolving these implicatures without specialized instruction-tuning.

The Recognition Action Disconnect

A striking finding in recent research is that models often "know" a statement is non-literal but cannot act on that knowledge. Akhila Yerukola and Maarten Sap identified a "generative gap": even when a model correctly identifies a speaker's ironic intent, it may still generate a response that treats the statement literally. This suggests that pragmatic reasoning is not a single "fact" the model learns, but a complex behavior it must integrate into its generation process.

Similarly, in task-oriented settings, Amogh Mannekote and Kristy Elizabeth Boyer found that while LLMs handle some nuances, smaller task-specific models suffer massive performance drops when users provide a justification (e.g., "I'm tired") instead of a direct command (e.g., "Set an alarm").

Grounding in Social and Physical Context

Pragmatics also requires Common Ground—the shared beliefs and physical context between speakers. Yanming Wan highlights this in the HandMeThat benchmark, where a robot must infer that "hand me that" refers to a knife because the human was just seen slicing fruit. Current models excel at labeling the knife but struggle to link the human's past actions to the current underspecified request.

In digital conversations, this manifests as "conversational friction." Rupak Sarkar and Neha Srikanth found that while LLMs can spot overt frustration, they struggle to detect the subtle, context-heavy misalignments in technical assumptions that lead to task failure.

Bridging the Gap with Recursive Reasoning

To solve these issues, researchers are moving beyond standard training. Linlu Qiu and Roger Levy suggest that models can be improved by incorporating the Rational Speech Act (RSA) framework. RSA treats communication as a recursive social game: the speaker chooses an utterance by imagining how a listener would interpret it, and the listener interprets an utterance by imagining why a speaker would choose it.

Furthermore, Shengguang Wu and Qi Su argue that Preference Optimization (PO) is more effective than standard fine-tuning for teaching models to navigate these "gray areas." By training on human preferences rather than just literal correctness, models can begin to align their outputs with the complex, non-literal expectations of human communication.

Papers in this version

Showing 60 of 142 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 156 papers
  2. Version 1 · 2026-10-04 · 142 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_pragmatic_inference_gap_c8323db9,
  title = {pragmatic inference gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pragmatic-inference-gap/rdr_c8323db9f36e4426bad564f7f2961913},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_challenge_of_pragmatic_reasoning_in__c8323db9_v1,
  title = {The Challenge of Pragmatic Reasoning in Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-challenge-of-pragmatic-reasoning-in-language-models/rdr_c8323db9f36e4426bad564f7f2961913/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}