Jointly Learning "What" and "How" from Instructions and Goal-States

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, Edward GrefenstetteWorkshops @ ICLR 2018 · OpenReview

Overview

AGILE (Adversarial Goal-Induced Learning from Examples) is a framework that allows artificial agents to learn how to follow instructions using only examples of successful "goal states," rather than relying on complex reward functions or step-by-step expert demonstrations. By decoupling the understanding of what a goal looks like from how to achieve it, the agent can generalize to new instructions and adapt to changes in environment physics more effectively than standard reinforcement learning.

The Challenge of Instruction Following

Teaching artificial agents to follow human instructions is notoriously difficult. Traditionally, researchers use two main approaches, both of which have significant drawbacks:

  1. Reinforcement Learning (RL): Requires a hand-coded reward function. For complex, ambiguous instructions (e.g., "Make the room look tidy"), writing a formal, programmatic reward function is often impossible.
  2. Imitation Learning (IL): Requires "expert trajectories"—step-by-step recordings of a human or expert performing the task. These are expensive to collect and often fail to transfer if the agent has a different body or action space than the expert.

The authors address the gap where we know what the end result should look like, but we cannot easily program a reward or record every single step required to get there.

The AGILE Framework

The authors introduce AGILE, a method inspired by Inverse Reinforcement Learning (IRL) and Generative Adversarial Imitation Learning (GAIL).

The AGILE framework architecture showing the interaction between the policy and discriminator.

The core innovation is training two neural networks simultaneously:

  • The Discriminator ($D_\phi$): This network learns the "What." Given an instruction and a state of the world, it predicts whether that state is a valid goal. It is trained using a dataset of expert-provided goal states (positive examples) and the agent's own failed attempts (negative examples).
  • The Policy ($\pi_\theta$): This network learns the "How." It tries to navigate the environment to reach a state that the Discriminator will label as a successful "goal."

Unlike GAIL, which looks at entire sequences of actions (trajectories), AGILE only evaluates the final state. This makes it much easier to provide training data, as a human only needs to provide a "photo" of the finished task rather than a video of the entire process.

Experimental Setup

The researchers evaluated AGILE in a 5x5 Gridworld environment. The grid contains an agent and various blocks featuring 3 shapes and 3 colors. To generate tasks, they used a functional programming-style language (GridLU-Relations) to create instructions involving five spatial relations (e.g., NorthFrom, SameLocation). An example instruction might be: "Put a red rectangle north of a blue object."

To test data efficiency, the team varied the number of (instruction, goal-state) pairs provided to the agent, ranging from $2^9$ to $2^{19}$ examples.

Key Insights

  • Decoupling "What" from "How": This is the paper's most powerful idea. Because the Discriminator and Policy are separate, the knowledge of what a "red square" is (held by the Discriminator) is independent of the motor skills needed to move it (held by the Policy).
  • Zero-Shot vs. Fine-Tuning: While the agent initially struggled to generalize to entirely new instructions, the authors found that the Discriminator's knowledge was "stickier" and more accurate than the Policy's. By freezing the Discriminator and only fine-tuning the Policy on new tasks, the agent improved significantly.
  • Adaptation "In the Wild": The authors simulated a "broken robot" scenario by making certain blocks immovable. Because the agent understood the goal (via the Discriminator), it could keep trying to find new ways to satisfy the instruction even when its old habits (the Policy) failed.

Performance and Results

Training performance of AGILE compared to RL baselines (left) and its data efficiency based on dataset size (right).
  • Overall Performance: AGILE achieved a 95% success rate, nearly matching a Reinforcement Learning baseline (99%) that had the "unfair" advantage of a perfect, hand-coded ground-truth reward function.
  • Data Efficiency: The agent began to succeed (surpassing a 50% success rate) with approximately 8,192 ($2^{13}$) examples. The authors suggest this volume of data is highly feasible to collect via crowd-sourcing.
  • Generalization: When faced with environment changes (immovable blocks), the agent's success rate initially dropped to 43%. However, by fine-tuning the policy while keeping the discriminator frozen, they recovered the success rate to 53%–67%, proving that the "What" knowledge is reusable and robust.

Limitations and Future Directions

  • Environment Complexity: The experiments were conducted in a minimalist 2D gridworld. Scaling this approach to complex 3D environments or continuous action spaces remains a significant challenge.
  • Discriminator Generalization: The Discriminator itself struggled to generalize to completely novel sub-expressions (like a color/shape combination it had never seen during training). Regularizing the discriminator could improve its ability to handle "out-of-distribution" instructions.
  • Instruction Ambiguity: While the paper mentions underspecified instructions, the gridworld tasks used fairly precise functional programs. Testing AGILE with truly messy, natural human language is a logical next step for future research.

Summary updated

Abstract

Training agents to follow instructions requires some way of rewarding them for behavior which accomplishes the intent of the instruction. For non-trivial instructions, which may be either underspecified or contain some ambiguity, it can be difficult or impossible to specify a reward function or obtain relatable expert trajectories for the agent to imitate. For these scenarios, we introduce a method which requires only pairs on instructions and examples of positive goal states, from which we can jointly learn a model of the instruction-conditional reward and a policy which executes instructions. Two sets of experiments in a gridworld compare the effectiveness of our method to that of RL when a reward function can be specified, and the application of our method when no reward function is defined. We furthermore evaluate the generalization of our approach to unseen instructions, and to scenarios where environment dynamics change outside of training, requiring fine-tuning of the policy ``in the wild''.
BibTeX
@inproceedings{bahdanau2018jointly,
  author = {Dzmitry Bahdanau and Felix Hill and Jan Leike and Edward Hughes and Pushmeet Kohli and Edward Grefenstette},
  title = {{Jointly Learning "What" and "How" from Instructions and Goal-States}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
  url = {https://openreview.net/forum?id=2fLWxHuC8b},
}
Figure