Abstract
We extend models for both following and generating natural language instructions by adding an explicit pragmatic layer. These pragmatics-enabled models explicitly reason about why speakers produce certain instructions, and about how listeners will react upon hearing them. Given learned base listener and speaker models, we build a pragmatic listener that uses the base speaker to reason counterfactually about alternative action descriptions, and a pragmatic speaker that uses the base listener to simulate the interpretation of candidate instruction sequences. Evaluation of language generation and interpretation in the SAIL navigation and SCONE instruction following datasets shows that the pragmatic inference procedure improves state-of-the-art listener models (at correctly interpreting human instructions) and speaker models (at producing instructions correctly interpretable by humans) in diverse settings.
Introduction
Standard AI models for instruction following (listeners) or instruction generation (speakers) typically operate on a literal level. A literal listener maps surface-level linguistic patterns to actions, while a literal speaker produces descriptions that are technically accurate but often ambiguous or unhelpful in context. However, human communication is inherently pragmatic; we don't just say what is true, we say what is useful. This paper introduces a unified framework for applying pragmatic inference to both generating and following natural language instructions in complex, sequential tasks.
The Problem: Literal vs. Pragmatic Agents
Literal agents often fail in environments where multiple paths or actions are technically consistent with a command. For instance, if a command is "walk forward four times," a literal listener might overshoot a target if the environment is ambiguous. Conversely, a literal speaker might provide that same vague command even when a more specific landmark-based instruction would be safer.
As shown in Figure 1, pragmatic reasoning allows a listener to resolve ambiguity by considering counterfactuals: "What else could the speaker have said if they meant a different goal?" Similarly, a pragmatic speaker acts as a mental simulator, running an internal model of the listener to identify when an instruction might be misinterpreted.
Methodology
The authors implement this recursive social reasoning using the Rational Speech Acts (RSA) framework. The system is built upon two base models:
- Base Models ($L_0$ and $S_0$): Standard neural sequence-to-sequence models. $L_0$ predicts actions from instructions, and $S_0$ predicts instructions from actions.
- Rational Speaker ($S_1$): Generates candidate instructions using $S_0$ and uses $L_0$ to simulate how a human might interpret them, selecting the one with the highest probability of success.
- Rational Listener ($L_1$): Generates candidate action sequences using $L_0$ and uses $S_0$ to determine which sequence most likely prompted the specific instruction given.
To maintain fluency, the rational models use a weighted combination of base model scores and pragmatic scores. Because the search space is large, the authors employ beam search to generate high-quality candidates for rescoring.
Evaluation and Datasets
The models were evaluated on four distinct domains requiring sequential reasoning:
- SAIL: A navigation task in a 2D grid of hallways with varied wall and floor patterns.
- SCONE: Three sub-domains involving object manipulation: Alchemy (beaker mixing), Scene (moving figures), and Tangrams (shape configuration).
Key Results
The pragmatic approach yielded significant improvements across all domains:
- Instruction Following: In the SAIL task, the rational listener improved accuracy by up to 10% over the base model, setting a new state-of-the-art.
- Instruction Generation: The gains were particularly massive in the Alchemy domain, where the percentage of instructions successfully followed by humans jumped from 29.3% (base) to 75.3% (rational).
- Surpassing Human Performance: In the Tangrams domain, the rational speaker’s instructions achieved a 92.7% success rate with human followers, significantly higher than the 66.0% success rate of the original human-generated instructions in the dataset. This suggests the model learned to systematically avoid common human ambiguities.
Conclusion
By incorporating an explicit layer of pragmatic inference, agents can effectively "think before they speak" or "read between the lines." This research demonstrates that modeling the communicative intent of others is a powerful tool for improving both the generation and following of instructions in complex environments. While computationally more expensive than literal models, the gains in task success and clarity represent a significant step toward more robust human-AI collaboration.
Summary updated