Abstract
Traditional AI assistants often focus on immediate instruction following, failing to capture the underlying preferences that drive user choices. This paper introduces a framework for reward inference from language in context. By modeling language as a pragmatic signal that reveals both desired actions and general preferences, the proposed model can generalize to novel environments. Using a new benchmark, FLIGHTPREF, the authors demonstrate that a pragmatic listener model significantly outperforms baselines in inferring long-term user rewards, bridging the gap between nearsighted execution and farsighted preference learning.
Introduction
The current paradigm for AI assistants primarily revolves around instruction following. While effective for executing specific tasks, this approach is often "nearsighted"—it lacks an understanding of the user's underlying reward function. For instance, if a user asks to book a specific flight, an agent that only follows instructions cannot determine if the choice was motivated by the airline, the price, or the departure time. Consequently, the agent cannot apply this knowledge to future, different scenarios.
This paper addresses the gap between immediate execution and long-term preference learning by treating language as a rich, pragmatic signal. The core insight is that language serves a dual purpose: it is both a pointer to a specific action in the current world and a window into the user’s internal values. By reasoning about why a user chose a specific utterance in a given context, an agent can infer a reward function that generalizes to entirely new environments.
Methods
The authors propose a pragmatic listener model based on the Rational Speech Acts (RSA) framework. The model assumes a rational speaker who chooses utterances to be informative about their goals. The system uses BERT-based encoders to represent language and MLPs to encode features and reward weights.
The Speaker Model ($S_1$)
The speaker model is decomposed into two primary objectives, balanced by a "nearsightedness" parameter ($\alpha$):
- Optimizing for Action ($p_{action}$): The speaker says something to ensure the agent picks the correct, optimal flight in the current context.
- Optimizing for Reward Descriptiveness ($p_{reward}$): The speaker chooses an utterance that accurately describes their general, long-term preferences ($\theta$).
The Rational Listener ($L_2$)
The AI agent acts as a rational listener. It uses Bayesian reasoning to invert the speaker model. When it observes an utterance in a specific context, it asks: "What reward function would most likely lead a rational speaker to say exactly that?" The model maintains a mathematical distribution (a posterior) of likely preferences and updates it after every interaction.
Datasets and Results
The authors introduced FLIGHTPREF, a multi-turn benchmark where a user with a hidden reward function guides an assistant to book flights across six rounds. The dataset contains 2,568 natural language utterances collected from human pairs.
The pragmatic model was evaluated on held-out accuracy—its ability to pick the correct flight in a brand-new environment based solely on inferred rewards.
- Superior Generalization: The full "Action + Reward" model achieved 59.1% accuracy, significantly outperforming models that only looked at actions (52.8%) or literal language meaning (57.8%).
- Efficiency: The model's performance with just a few sentences approached that of an "Oracle" model with perfect knowledge of several user preferences. This suggests that pragmatics can effectively resolve ambiguity; for example, if a user specifies an airline that happens to be the most expensive option, the model can infer that airline loyalty is a high-priority reward feature.
Discussion and Future Work
While the model shows significant promise, it currently assumes linear reward functions and operates in a relatively simple action space. Real-world human preferences are often non-linear and more complex. Future work involves scaling this approach to open-ended assistants and complex robotics. Additionally, the authors suggest moving toward interactive learning, where the agent actively asks clarifying questions to resolve uncertainty in its reward posterior. This research highlights that language is not just a command but a critical tool for aligning AI agents with human values.
Summary updated