Learning to Understand Goal Specifications by Modelling Reward

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Seyed Arian Hosseini, Pushmeet Kohli, Edward Grefenstette

Introduction

Recent advancements in deep reinforcement learning (RL) have demonstrated that agents can learn to follow language-like instructions using environment rewards. However, this traditional approach places a heavy burden on environment designers, who must manually program complex, language-conditional reward functions. As environments scale in complexity and natural language instructions become more ambiguous or underspecified (e.g., "arrange these blocks in an L-shape"), writing robust reward functions becomes an intractable engineering challenge.

To address this limitation, researchers from Mila and DeepMind introduced AGILE (Adversarial Goal-Induced Learning from Examples). AGILE is a framework that allows RL agents to learn complex language instructions without hard-coded reward functions. By jointly training a policy alongside a reward model that learns to recognize "success" from expert goal-state examples, agents can master tasks ranging from spatial relations to abstract shape building.

The AGILE Framework

The core objective of the paper is to remove the "onus on the designer" by allowing the agent to induce the meaning of instructions directly from examples of completed tasks. This effectively separates the "what" (the goal representation) from the "how" (the execution policy).

The AGILE framework replaces hard-coded environment rewards with a learned reward model trained on expert goal states.

AGILE draws inspiration from Generative Adversarial Imitation Learning (GAIL) but focuses strictly on goal states rather than full expert trajectories. The framework consists of two main components:

  1. The Reward Model (Discriminator): A neural network is trained to distinguish between "positive" examples (a dataset of expert-provided goal states paired with instructions) and "negative" examples (states the agent actually visits during training).
  2. The Policy: An RL agent (using the A3C algorithm) is trained to maximize the reward provided by this discriminator.

A key challenge in this adversarial setup is handling false negatives. As the agent improves, it reaches the true goal more frequently. If the discriminator treats all agent-visited states as "negative," it will eventually start penalizing successful task completion. To mitigate this, the authors introduce a heuristic: they rank the agent's states by the discriminator's current score and discard the top $\rho$ percent (the "anticipated negative rate") to avoid training the discriminator on likely successes.

For the neural architecture, the researchers tested two models to process instructions: Neural Module Networks (NMN), which parse instructions into a tree of neural modules, and FiLM-LSTM, a structure-agnostic approach using Feature-wise Linear Modulation to condition a CNN on an LSTM-encoded instruction.

Datasets and Environments

To evaluate AGILE, the researchers introduced GridLU (Grid Language Understanding), a $5 \times 5$ grid world where an agent can pick up and move colored shapes.

Examples of the GridLU-Arrangements task, where agents must build specific abstract shapes.

They evaluated the framework on two distinct tasks within this environment:

  • GridLU-Relations: Involves 990 possible instructions dictating spatial relationships (e.g., "Put a red circle north of a blue square").
  • GridLU-Arrangements: A highly abstract task where the agent must build specific shapes like "Eels," "Snakes," or "L-shapes." This task is massive, encompassing roughly 390 million possible valid goal states.

Key Insights and Results

AGILE demonstrated strong performance and generalization capabilities across both tasks:

  • High Success Rates: On the Relations task, AGILE achieved a 99.5% success rate, matching the performance of agents trained with hand-coded rewards and specialized auxiliary tasks. In the vastly more complex Arrangements task, the agent achieved a 58% success rate overall (and 92% on simpler 4-block tasks) using only 100,000 expert examples—representing less than 0.03% of the total possible goal states.
  • Perfect Generalization: When 10% of instructions were held out during training, the reward model generalized perfectly. This proves the model learned the underlying semantics of the language rather than merely memorizing state-instruction pairs.
  • The "Natural Curriculum": Interestingly, AGILE-trained agents often learned faster than those with ground-truth rewards. The authors hypothesize that early in training, the imperfect reward model is "generous" with false positives. This provides a dense reward signal that encourages effective exploration, acting as a naturally emerging curriculum.
  • Reusability: Because the reward model independently understands the goal, it can be frozen and reused. If the environment's physics change (e.g., a specific block becomes immovable), the agent can be fine-tuned using the existing reward model to find novel ways to achieve the same goal without requiring new expert data.

Limitations and Future Work

While AGILE presents a significant step forward, the authors note several limitations. The current work relies on a 2D "God's eye view" of a grid. Transitioning to 3D first-person environments would require the agent to handle occlusions and distinguish its own embodiment from the environment. Additionally, the "anticipated negative rate" ($\rho$) is a task-specific hyperparameter that requires careful tuning; if set incorrectly, it can either degrade the reward model's accuracy or deprive the agent of informative training data. Finally, while the tested models handled constrained grammar well, scaling this approach to truly unconstrained natural language remains a significant frontier for future research.

Summary updated

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.
BibTeX
@inproceedings{bahdanau2019learning,
  author = {Dzmitry Bahdanau and Felix Hill and Jan Leike and Edward Hughes and Seyed Arian Hosseini and Pushmeet Kohli and Edward Grefenstette},
  title = {{Learning to Understand Goal Specifications by Modelling Reward}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=pWDyE_XzJ9},
}
Figure