Inferring Rewards from Language in Context

Jessy Lin, Daniel Fried, Dan Klein, Anca D. Dragan

Abstract

Traditional AI assistants often focus on immediate instruction following, failing to capture the underlying preferences that drive user choices. This paper introduces a framework for reward inference from language in context. By modeling language as a pragmatic signal that reveals both desired actions and general preferences, the proposed model can generalize to novel environments. Using a new benchmark, FLIGHTPREF, the authors demonstrate that a pragmatic listener model significantly outperforms baselines in inferring long-term user rewards, bridging the gap between nearsighted execution and farsighted preference learning.

Introduction

The current paradigm for AI assistants primarily revolves around instruction following. While effective for executing specific tasks, this approach is often "nearsighted"—it lacks an understanding of the user's underlying reward function. For instance, if a user asks to book a specific flight, an agent that only follows instructions cannot determine if the choice was motivated by the airline, the price, or the departure time. Consequently, the agent cannot apply this knowledge to future, different scenarios.

Overview of the reward inference process. The agent observes the user's choice and language in a specific context to update its belief about the user's underlying reward function.

This paper addresses the gap between immediate execution and long-term preference learning by treating language as a rich, pragmatic signal. The core insight is that language serves a dual purpose: it is both a pointer to a specific action in the current world and a window into the user’s internal values. By reasoning about why a user chose a specific utterance in a given context, an agent can infer a reward function that generalizes to entirely new environments.

Methods

The authors propose a pragmatic listener model based on the Rational Speech Acts (RSA) framework. The model assumes a rational speaker who chooses utterances to be informative about their goals. The system uses BERT-based encoders to represent language and MLPs to encode features and reward weights.

The Speaker Model ($S_1$)

The speaker model is decomposed into two primary objectives, balanced by a "nearsightedness" parameter ($\alpha$):

  1. Optimizing for Action ($p_{action}$): The speaker says something to ensure the agent picks the correct, optimal flight in the current context.
  2. Optimizing for Reward Descriptiveness ($p_{reward}$): The speaker chooses an utterance that accurately describes their general, long-term preferences ($\theta$).
The speaker model components: p_action represents the intent to guide an action, while p_reward represents the intent to describe general preferences.

The Rational Listener ($L_2$)

The AI agent acts as a rational listener. It uses Bayesian reasoning to invert the speaker model. When it observes an utterance in a specific context, it asks: "What reward function would most likely lead a rational speaker to say exactly that?" The model maintains a mathematical distribution (a posterior) of likely preferences and updates it after every interaction.

Datasets and Results

The authors introduced FLIGHTPREF, a multi-turn benchmark where a user with a hidden reward function guides an assistant to book flights across six rounds. The dataset contains 2,568 natural language utterances collected from human pairs.

The pragmatic model was evaluated on held-out accuracy—its ability to pick the correct flight in a brand-new environment based solely on inferred rewards.

  • Superior Generalization: The full "Action + Reward" model achieved 59.1% accuracy, significantly outperforming models that only looked at actions (52.8%) or literal language meaning (57.8%).
  • Efficiency: The model's performance with just a few sentences approached that of an "Oracle" model with perfect knowledge of several user preferences. This suggests that pragmatics can effectively resolve ambiguity; for example, if a user specifies an airline that happens to be the most expensive option, the model can infer that airline loyalty is a high-priority reward feature.

Discussion and Future Work

While the model shows significant promise, it currently assumes linear reward functions and operates in a relatively simple action space. Real-world human preferences are often non-linear and more complex. Future work involves scaling this approach to open-ended assistants and complex robotics. Additionally, the authors suggest moving toward interactive learning, where the agent actively asks clarifying questions to resolve uncertainty in its reward posterior. This research highlights that language is not just a command but a critical tool for aligning AI agents with human values.

Summary updated

Abstract

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).
BibTeX
@inproceedings{lin2022inferring,
  author = {Jessy Lin and Daniel Fried and Dan Klein and Anca D. Dragan},
  title = {{Inferring Rewards from Language in Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=WIaHMkv9c1y},
}
Figure