Learning to Understand Goal Specifications by Modelling Reward

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, Edward Grefenstette

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations—and for instructions—not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

Introduction

In traditional Reinforcement Learning (RL), training an agent to follow an instruction like "put the red block north of the blue square" requires a manually defined reward function. This function acts as a hard-coded logic gate, granting points only when specific spatial conditions are met. However, as instructions become more complex, ambiguous, or abstract (e.g., "build something that looks like an 'L'"), writing these functions becomes a significant engineering bottleneck. It requires designers to perfectly translate human linguistic nuances into rigid code.

This research introduces AGILE (Adversarial Goal-Induced Learning from Examples), a framework designed to bypass the need for hand-coded rewards by learning the reward function directly from examples of successful outcomes.

Methodology

The AGILE framework draws inspiration from Generative Adversarial Networks (GANs) and Inverse Reinforcement Learning. It consists of two neural networks trained in tandem:

  1. The Reward Model (Discriminator): This model is trained to distinguish between "true" goal states (from a static dataset of expert examples) and "fake" states (the outcomes produced by the agent). It learns to map language instructions to their corresponding successful final states.
  2. The Policy (Generator): This is the RL agent that interacts with the environment. Its objective is to maximize the reward signal provided by the Reward Model.
The AGILE policy training loop, where the agent receives rewards from a learned model rather than a hard-coded environment function.

A key innovation in AGILE is the use of Neural Module Networks (NMN). To handle the compositional nature of language, the models assemble a custom neural architecture for each instruction based on its linguistic structure. Additionally, to prevent training collapse as the agent improves, the authors implemented a heuristic to discard high-scoring agent states from the negative training pool, assuming they might be "false negatives" (actual successes).

Experimental Setup: GridLU

The framework was evaluated in GridLU (Grid Language Understanding), a 5x5 grid world containing blocks of various colors and shapes. Two primary benchmarks were used:

  • GridLU-Relations: Focused on spatial reasoning using five relationships (e.g., NorthFrom, SameLocation).
  • GridLU-Arrangements: A more abstract task requiring the agent to build shapes like "Eel," "Snake," or "Triangle." These shapes are orientation-invariant and can be constructed anywhere on the grid, making them significantly harder to define with traditional code.
Examples of GridLU tasks, including spatial relations and abstract arrangements like 'Eel' or 'Snake'.

Results and Discussion

The experiments yielded several key insights:

  • Separation of "What" and "How": Decoupling the reward model from the policy allows the system to learn the semantics of language independently of the physics of the environment.
  • Emergent Curriculum: AGILE-trained agents often learned faster than those with ground-truth rewards. The reward model's early "leniency" provides a denser, more informative signal than a sparse binary reward.
  • Generalization: The system demonstrated strong zero-shot capabilities. An agent trained on 90% of possible instructions performed equally well on the held-out 10%, indicating a deep understanding of linguistic composition.

On the Relations task, AGILE achieved a 99.5% success rate. On the complex Arrangements task—where no ground-truth reward function was even written—the agent succeeded 58% of the time (rising to 92% for simpler 4-block tasks). Furthermore, when environment physics were altered (e.g., making certain blocks immovable), the agent successfully adapted by fine-tuning the policy while keeping the reward model frozen, with success rates recovering to 69.3%.

Conclusion

AGILE demonstrates that reward models trained on goal states can effectively replace hand-coded reward functions for complex language-conditioned tasks. While currently limited to 2D environments and requiring substantial expert data (approx. 100,000 examples), the framework provides a scalable path toward training agents in more complex 3D worlds where manual reward specification is intractable. Future work will focus on improving data efficiency and handling higher-dimensional state spaces.

Summary updated

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.
BibTeX
@inproceedings{bahdanau2019learning,
  author = {Dzmitry Bahdanau and Felix Hill and Jan Leike and Edward Hughes and Arian Hosseini and Pushmeet Kohli and Edward Grefenstette},
  title = {{Learning to Understand Goal Specifications by Modelling Reward}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=pWDyE_XzJ9},
}
Figure