Inferring Rewards from Language in Context

Jessy Lin, Daniel Fried, Dan Klein, Anca D. Dragan

Abstract

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).

Introduction

In the development of AI assistants, there is a persistent divide between Instruction Following (IF) and Inverse Reinforcement Learning (IRL). IF models are typically "nearsighted": if a user says, "Book the JetBlue flight," the model performs the action but fails to learn that the user generally prefers JetBlue for future travel. Conversely, IRL models attempt to learn a user’s underlying "reward function" by observing their actions, but they often ignore the rich, explicit information contained in natural language.

This paper bridges this gap by treating language as a dual-purpose signal. Language is not just a way to trigger an immediate action; it is a pragmatic tool used by humans to reveal their long-term preferences. By inferring these rewards from situated context, agents can generalize to new, unseen environments far more effectively than standard models.

Pragmatic Reward Inference

The authors propose a model based on the Rational Speech Acts (RSA) framework. The core of the approach is a "Rational Listener" ($L_2$) that performs Bayesian updates on a reward function ($\theta$) after observing an utterance ($u$) in a specific context ($M$).

The model assumes a Speaker ($S_1$) who chooses words based on two competing goals:

  1. Action Optimization ($p_{action}$): Choosing words that help the agent pick the correct item in the current round.
  2. Reward Descriptiveness ($p_{reward}$): Choosing words that accurately describe their general, long-term preferences.

A "nearsightedness" parameter ($\alpha$) balances these goals. If $\alpha$ is high, the model assumes the user is simply trying to complete the current task; if low, it assumes the user is attempting to teach the agent their general preferences. By reasoning about the alternative utterances a user could have chosen, the model performs pragmatic inference to determine which features (e.g., price, carrier, stops) are most important to the user.

The FLIGHTPREF Benchmark

To test this framework, the authors introduced FLIGHTPREF, a multi-turn flight-booking dataset:

  • The Task: A user with a hidden reward function gives natural language instructions to an assistant across multiple rounds.
  • Data: 2,568 utterances collected from 813 games played by human workers.
  • Generalization: Unlike standard datasets, FLIGHTPREF requires the agent to use language from past interactions to make optimal decisions in entirely new flight sets where the previous instructions no longer apply directly.

Key Insights

  • Language as a Salience Filter: While actions provide evidence about all features of a choice, language tends to highlight the most "salient" preferences. If a user mentions "JetBlue," it is a strong signal that the carrier is a high-weight feature in their reward function.
  • Pragmatic Reasoning: The model doesn't just process keywords; it reasons about context. If a user says "the JetBlue flight" when a cheaper flight is available, the model infers that carrier preference outweighs price sensitivity for that user.
  • Complementary Signals: Language is most useful early in an interaction to quickly narrow down major preferences. Actions become more useful over time for refining the "fine-grained" trade-offs that are difficult to describe in words.

Results and Performance

The pragmatic model was evaluated on its ability to pick the user's preferred flight in brand-new environments:

  • Joint Model Superiority: The "Action + Reward" model achieved 59.1% accuracy, significantly outperforming models that only looked at actions (52.8%) or only treated language as a description (57.8%).
  • Relative Improvement: The full model showed a 12% relative improvement over models that treat language only as a mapping to immediate actions.
  • Learning Curves: As the number of observed utterances increased, the joint model's distance to the true reward function decreased more sharply than baseline models, proving it effectively "learns" the user over time.

Limitations and Future Work

The study currently utilizes linear reward functions and a constrained feature space. Real-world preferences are often non-linear and more complex. Furthermore, the model assumes a uniform prior over rewards, whereas humans typically hold common-sense priors (e.g., preferring lower prices). Future research could incorporate these priors or move toward active learning, where the agent asks clarifying questions to resolve reward uncertainty.

Summary updated

Abstract

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).
BibTeX
@inproceedings{lin2022inferring,
  author = {Jessy Lin and Daniel Fried and Dan Klein and Anca D. Dragan},
  title = {{Inferring Rewards from Language in Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=WIaHMkv9c1y},
}
Figure