Pragmatic Language Generation via Listener Simulation

35 concepts26 papers1 proposal

Imagine looking at a grid of three chairs: a standard wooden chair, a blue plastic chair, and a blue rocking chair. If you want someone to select the blue rocking chair, describing it simply as "a chair" or even "a blue chair" is factually true but communicatively useless. To ensure success, you must say "the rocking chair."

Standard language models excel at being literally true—they predict tokens that accurately describe data based on training likelihood. However, they often fail at being pragmatically informative because they lack an understanding of the listener's context. To bridge this gap, a prominent research direction focuses on pragmatic speaker modeling. The core intuition is that to communicate effectively, an AI must internally simulate how a listener will interpret its words before it actually speaks.

The Core Mechanism: The Internal Feedback Loop

The foundational architecture of a pragmatic speaker relies on a communicative feedback loop. Instead of directly outputting text, the speaker acts as a dual-agent system containing two components:

  1. A Base Speaker (or proposal model) that generates candidate descriptions or instructions.
  2. An Internal Listener (or comprehension model) that evaluates how well those candidates achieve the intended goal.

This concept traces back to early symbolic systems like ANTLIMA, developed by Jörg R. J. Schirra et al. in ANTLIMA - A Listener Model with Mental Images. The authors evaluated planned natural language descriptions by simulating the "mental images" they would trigger in a listener. If the simulated visualization was ambiguous or contradicted the world state, the system re-entered the selection phase to find a more precise description.

In modern deep learning, this is formalized using frameworks like Rational Speech Acts (RSA). As demonstrated by Jacob Andreas et al. in Reasoning about Pragmatics with Neural Listeners and Speakers, a reasoning agent can generate a pool of candidate descriptions using a standard generative model and then select the optimal output by simulating how a neural listener model would interpret each candidate. This allows the system to produce context-aware descriptions that distinguish a target from distractors without requiring explicit pragmatic training data.

[ Context / Distractors ]
           │
           ▼
   ┌───────────────┐     Candidates     ┌───────────────────┐
   │ Base Speaker  │───────────────────>│ Internal Listener │
   └───────────────┘                    └───────────────────┘
           ▲                                      │
           │          Re-rank / Select            │
           └──────────────────────────────────────┘

Pragmatic Selection: Global Reranking vs. Incremental Decoding

How does this internal simulation happen during text generation? Researchers have converged on two primary paradigms: global reranking and incremental decoding.

Global Reranking

In global reranking, the generation process is split into a distinct search phase and a selection phase, an approach analyzed by Lingjun Zhao et al. in Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models. First, the base speaker uses a method like beam search to generate a set of complete candidate utterances. Second, the internal listener scores each candidate based on its discriminative utility.

Incremental Pragmatic Decoding

Evaluating full sentences after they are generated can be computationally expensive and cannot fix mistakes made early in the generation process. To solve this, Noah Goodman et al. in Pragmatically Informative Image Captioning with Character-Level Inference introduced incremental pragmatic decoding.

Here, the listener simulation happens token by token. The model selects the next token by balancing its literal probability with its immediate discriminative utility, ensuring the utterance remains informative from the very first word.

Expanding the Scope: Beyond Reference Games

While reference games provide a clean mathematical formulation for pragmatics, the listener-aware paradigm has been successfully applied to broader, more complex language tasks.

Machine Translation

Standard translation models often suffer from meaning collapse, translating distinct source sentences into the same ambiguous target sentence. In Lost in Machine Translation: A Method to Reduce Meaning Loss, Reuben Cohn-Gordon et al. integrate pragmatic reasoning into translation decoding by using a back-translation model as a simulated listener. The system selects target sentences that allow the listener to accurately reconstruct the original source sentence, preserving subtle nuances.

Model Calibration and Trust

Large language models often express high confidence even when they are incorrect. To combat this, Elias Stengel-Eskin et al. developed LACIE in LACIE: Listener-Aware Finetuning for Calibration in Large Language Models. By simulating a two-agent game where a listener decides whether to accept an answer based on perceived confidence, the authors optimize the speaker's tone and explicit confidence markers to align with its actual accuracy, teaching the model to naturally hedge or abstain when uncertain.

Managing Uncertainty

When a speaker encounters an object from an unknown category, a naive model might guess a misleading label. In Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories, Sina Zarrieß et al. show that a pragmatic speaker integrates feedback from a pragmatic listener to prioritize general attributes (like color or location) over uncertain category names. This mirrors human behavior, as explored by Amir Sadovnik et al. in It's Not Polite to Point: Describing People with Uncertain Attributes, where explicitly modeling the listener's identification process under visual uncertainty achieves human-level accuracy in generating referring expressions.

Collaborative Dialogue and Theory of Mind

In interactive settings, a speaker must adapt to a partner's changing perspective or capabilities. Daniel Fried et al. in Reference-Centric Models for Grounded Collaborative Dialogue show that using an internal reference resolver as a proxy for a dialogue partner significantly improves common grounding.

Taking this further, Hao Zhu et al. in Few-shot Language Coordination by Modeling Theory of Mind explicitly model a partner's mental state using Theory of Mind (ToM). This allows a speaker to dynamically switch between high-level and low-level instructions based on simulated partner reactions.

From an evolutionary perspective, Marieke Woensdregt et al. in A model of cultural co-evolution of language and perspectivetaking demonstrate that this pressure to model a listener's perspective is what drives the emergence of informative linguistic conventions in the first place. Similarly, Andy Liu et al. in Computational Language Acquisition with Theory of Mind show that equipping agents with an internal model that predicts a listener's mental state significantly improves their ability to acquire language under environmental pressure.

Optimizing the Bottleneck: Joint Inference and Amortization

Running a full listener simulation at inference time introduces significant computational overhead. To address this, researchers have explored ways to integrate listener feedback directly into the training phase or tightly couple the two modules:

  • Amortized Pragmatics: Instead of performing expensive recursive reasoning at runtime, Julia White et al. in Learning to refer informatively by amortizing pragmatic reasoning demonstrate that incorporating an internal listener model during training provides a highly effective gradient for learning pragmatic speech. This allows a neural speaker to directly optimize for listener success, achieving the benefits of RSA reasoning at the speed of a standard captioner.
  • Pragmatic Joint Inference: Mustafa Omer Gul et al. in CoGen: Learning from Feedback with Coupled Comprehension and Generation show that tightly coupling comprehension (listening) and generation (speaking) capabilities allows agents to share data and perform joint inference, leading to more precise descriptions and more human-like language diversity.

Ultimately, listener-aware speaker modeling shifts the objective of language generation from surface-level imitation to functional communication. Whether optimizing for visual discriminability using chart-based algorithms (Nikolaos Engonopoulos et al. in Generating effective referring expressions using charts) or navigating complex environments, simulating the listener ensures that AI systems do not just produce text, but are truly understood.

Key Foundational Work

A Game-Theoretic Approach to Generating Spatial Descriptions (2010) — contributed 1 concept elements to this direction.
ANTLIMA - A Listener Model with Mental Images (1993) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pragmatic_speaker_modeling_c8e67302,
  title = {pragmatic speaker modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pragmatic-speaker-modeling/rdr_c8e673029f364df39c75b8cad5a0443d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_pragmatic_speaker_modeling_c8e67302_v1,
  title = {pragmatic speaker modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pragmatic-speaker-modeling/rdr_c8e673029f364df39c75b8cad5a0443d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake