Are LLMs good pragmatic speakers?

Mingyue Jian, Siddharth Narayanaswamy

Introduction

Large Language Models (LLMs) have demonstrated remarkable "formal linguistic competence," excelling at grammar, syntax, and local coherence. However, their "functional competence"—the ability to use language effectively in context—remains a subject of intense research. A core component of functional competence is pragmatics: the art of being informative without being redundant.

While a literal speaker might describe a "small red chair" in a room full of chairs, a pragmatic speaker recognizes that if only one chair is small, "the small one" is the most efficient and helpful description. This study investigates whether LLMs like Llama3-8B-Instruct inherently possess this pragmatic reasoning or if they primarily function as "truth-tellers" that prioritize literal accuracy over contextual informativeness.

Methodology: The Rational Speech Act Framework

To evaluate pragmatic behavior, the researchers employed the Rational Speech Act (RSA) framework. RSA formalizes communication as a recursive social process between two agents:

  1. Literal Listener ($L_0$): An agent that interprets utterances based strictly on their logical truth conditions.
  2. Pragmatic Speaker ($S_1$): An agent that selects utterances to maximize the probability that the Literal Listener identifies the correct target, while simultaneously minimizing the "cost" (typically utterance length).

The study utilized the TUNA corpus, a benchmark dataset for Referring Expression Generation (REG) focused on the furniture domain (chairs, desks, sofas, fans). Objects in TUNA are defined by discrete attributes such as color, size, and orientation. The researchers generated 2,940 "Reference Games" where the model had to describe a target object among distractors, resulting in over 386,000 utterance instances for evaluation.

Key Insights: Literal Bias and Ranking Struggles

A primary finding is the "Literal Bias" exhibited by LLMs. Llama3 tended to assign higher probabilities to exhaustive, feature-heavy descriptions. While these descriptions are "true," they lack the concise informativeness favored by human-like pragmatic models.

Scatter plot comparing Vanilla LLM probabilities against RSA model probabilities, illustrating the literal bias and variance in alignment.

The research also distinguished between "coarse" and "fine" ranking. The LLM proved capable of distinguishing a "good" (true) utterance from a "bad" (false) one across the entire possible utterance space. However, it struggled significantly when tasked with ranking the best pragmatic options against each other. This suggests that while the model understands truth conditions, it does not yet capture the nuances of pragmatic preference.

Quantitative Results

The alignment between Llama3 and the RSA model was measured using Pearson Correlation Coefficients (PCC) and Spearman’s Rank Correlation Coefficients (SRCC).

Distribution of Pearson and Spearman correlation coefficients across reference games, showing the performance gap in ranking top-k candidates.
  • Weak Overall Correlation: The PCC between LLM and RSA scores hovered around 0.3, indicating a positive but modest relationship.
  • Ranking Failure in Top-k: When evaluating only the "Top-k" pragmatic utterances (the most likely candidates), the SRCC dropped precipitously to between 0.059 and 0.086. This confirms that the LLM’s internal ranking of high-quality descriptions does not align with pragmatic theory.
  • Meaning Function Sensitivity: The RSA model's performance was highly dependent on the "meaning function" used to determine truth. Rule-based functions outperformed prompt-based ones, suggesting that LLMs struggle with consistent internal logic when judging the truth of their own descriptions.

Conclusion and Future Directions

The study concludes that Llama3-8B-Instruct is a proficient "truth-teller" but a poor "pragmatist." It prioritizes literal accuracy over the efficient, context-aware communication defined by the RSA framework.

Future work should explore whether larger models (e.g., Llama-70B or GPT-4) exhibit emergent pragmatic capabilities and whether iterative, multi-turn interactions can improve alignment through feedback loops. Additionally, incorporating human baselines will be essential to determine if RSA-ideal utterances truly reflect human communicative preferences better than current LLM outputs.

Summary updated

BibTeX
@inproceedings{jian2024are,
  author = {Mingyue Jian and Siddharth Narayanaswamy},
  title = {{Are LLMs good pragmatic speakers?}},
  booktitle = {Behavioral ML @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=htFtNdgWdf},
}
Figure