Abstract
Current large language models (LLMs) have demonstrated emerging capabilities in social intelligence tasks, including implicature resolution and theory-of-mind reasoning, both of which require substantial pragmatic understanding. However, how LLMs acquire this pragmatic competence throughout the training process remains poorly understood. In this work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of alternatives, to evaluate whether LLMs at different training stages can accurately infer nuanced speaker intentions. Each instance pairs two equally plausible yet pragmatically divergent continuations and requires the model to (i) infer the speaker's intended meaning and (ii) explain when and why a speaker would choose one utterance over its alternative, thus directly probing pragmatic competence through contrastive reasoning. We systematically evaluate 22 LLMs across 3 key training stages: after pre-training, supervised fine-tuning (SFT), and preference optimization, to examine the development of pragmatic competence. Our results show that even base models exhibit notable sensitivity to pragmatic cues, which improves consistently with increases in model and data scale. Additionally, SFT and RLHF contribute further gains, particularly in cognitive-pragmatic scenarios. These findings highlight pragmatic competence as an emergent and compositional property of LLM training and offer new insights for aligning models with human communicative norms.
Introduction
While Large Language Models (LLMs) have shown surprising "social intelligence," it has remained unclear exactly when these models learn to read between the lines—a skill known as pragmatic competence. Previous research suggested that pragmatic abilities only appear after instruction tuning, but those studies often relied on simple binary tasks and proprietary models with opaque training histories. This research investigates the "black box" of pragmatic development, asking whether models learn social nuance from massive raw text (pre-training) or require explicit human alignment (SFT/DPO).
Methodology and the ALTPRAG Dataset
The researchers developed a systematic evaluation framework using 22 different LLMs at various stages of development. Central to this is the ALTPRAG dataset, which consists of 1,300 datapoints grounded in the linguistic concept of "alternatives"—the infinite space of what a speaker could have said but didn't.
The evaluation process involves:
- Free-Form Probing: Models generate natural language explanations of a speaker's intent and the situational motivations for choosing one response over another.
- LLM-as-a-Judge: GPT-4o evaluates these explanations using a 10-point accuracy score and pairwise win rates.
- Three-Dimensional Analysis: Pragmatic wins are categorized into Cognitive-pragmatic (inferring goals), Pragmalinguistic (rhetorical strategies), and Sociopragmatic (social norms).
Key Insights
- Pragmatics as an Emergent Property of Scale: Base models (pre-alignment) already show significant pragmatic competence. A 7B-8B parameter base model typically scores around 6/10.
- The Power of Pre-training Data: The volume of pre-training data is a massive factor. For instance, the Qwen-3 1.7B model (trained on 36 trillion tokens) outperformed the OLMo-2 7B model, suggesting that exposure to vast amounts of human dialogue is a primary driver of social learning.
- SFT vs. DPO Roles: There is a "division of labor" in alignment. SFT primarily boosts cognitive-pragmatic competence, while DPO (Direct Preference Optimization) significantly improves sociopragmatic awareness.
Results and Performance
The study demonstrates consistent scaling across all model families (OLMo, LLaMA, Qwen). Pragmatic scores improved as models moved from Base to SFT to DPO.
Notably, LLaMA-3.1-70B-DPO achieved the highest average score of 9.28/10, showing near-human alignment. Furthermore, modern base models like OLMo-2 32B now outperform older, fully-tuned models like GPT-3-175B on classic pragmatic benchmarks, even without instruction tuning.
Limitations and Future Work
The authors acknowledge several constraints, including the need for more granular categorization of phenomena like humor or irony and the potential for shared architectural biases among tested models. Future work will involve expanding ALTPRAG to include more diverse cultural contexts and testing how these pragmatic abilities translate into actual conversational performance.
Summary updated