Learning to Follow Language Instructions with Adversarial Reward Induction

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, Edward Grefenstette

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations—and for instructions—not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

Introduction

In traditional Reinforcement Learning (RL), environment designers must manually engineer reward functions—code that explicitly defines success for every possible instruction. While simple commands like "go to the red square" are easy to program, natural language is often underspecified and ambiguous. For complex tasks like "build an L-like shape," hard-coding every valid orientation and position becomes a significant bottleneck.

To address this, we introduce the AGILE (Adversarial Goal-Induced Learning from Examples) framework. AGILE allows agents to induce the meaning of a goal specification by observing examples of successful outcomes, rather than relying on manually programmed reward logic.

The AGILE Framework

AGILE consists of two primary components trained in an adversarial loop: a Reward Model (Discriminator) and a Policy (Actor).

The AGILE framework: (a) Policy training via reward model feedback; (b) Reward model training using expert goal states (A) vs. agent-generated states (B).
  1. The Reward Model: This model learns to distinguish between "positive" examples (a dataset of expert-provided {instruction, goal-state} pairs) and "negative" examples (states visited by the agent during training). Unlike previous methods that require full demonstrations of how to perform a task, AGILE only requires the final result.
  2. The Policy: The RL agent interacts with the environment to maximize the reward provided by the current Reward Model.

To handle the compositional nature of language, we utilize Neural Module Networks (NMN) and FiLM (Feature-wise Linear Modulation) layers to condition the agent's visual processing on textual instructions. A key innovation is a ranking heuristic that discards agent-generated states that the reward model identifies as likely goals, preventing the training signal from collapsing as the agent improves.

Experimental Evaluation

We evaluated AGILE in GridLU (Grid Language Understanding), a custom 2D grid world featuring two primary task categories:

  • GridLU-Relations: Requires understanding spatial logic (e.g., "Put a red circle north of a blue square"), involving 990 possible instructions.
  • GridLU-Arrangements: An abstract task where agents arrange blocks into shapes like "Eel" or "Snake." This task features a massive state space of approximately 390 million possible goal states, of which the agent sees only a tiny fraction (0.03%) as expert examples.
Example of a spatial relation task in the GridLU environment: "Green triangle west of a red circle".

Results and Discussion

AGILE demonstrated superior performance and adaptability across both tasks:

  • Performance: On the Relations task, AGILE achieved a 99.5% success rate, outperforming standard agents trained with ground-truth rewards. In the complex Arrangements task, it reached a 58% success rate with limited expert data.
  • Emergent Curriculum: AGILE-trained agents often learned faster than those given "perfect" rewards. The reward model acts as a curriculum, providing denser, "close enough" rewards early in training compared to sparse binary success signals.
  • Adaptability: When environment physics were altered (e.g., making specific blocks immovable), the agent successfully adapted its strategy by reusing the learned reward model. This proves that AGILE effectively decouples the understanding of "what" a goal is from the execution of "how" to reach it.

Conclusion

AGILE provides a scalable path for RL agents to follow complex, natural language instructions without the need for exhaustive reward engineering. By learning from goal-state examples, agents can generalize linguistic concepts and adapt to environmental changes, though future work is needed to extend this framework to procedural commands and high-dimensional 3D environments.

Summary updated

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.
BibTeX
@inproceedings{bahdanau2019learning,
  author = {Dzmitry Bahdanau and Felix Hill and Jan Leike and Edward Hughes and Pushmeet Kohli and Edward Grefenstette},
  title = {{Learning to Follow Language Instructions with Adversarial Reward Induction}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=pWDyE_XzJ9},
}
Figure