From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Justin Fu, Anoop Korattikara, Sergey Levine, Sergio Guadarrama

Introduction

Reinforcement learning (RL) agents are typically trained to optimize a reward function. However, defining these functions for complex, vision-based tasks—such as "move the cup from the sink to the kitchen table"—is a significant challenge. While many approaches attempt to map language instructions directly to sequences of actions (policies), these models often fail to generalize to new environments. This paper proposes a shift in perspective: learning a language-conditioned reward function rather than a policy. By grounding language as a goal state rather than a set of moves, agents can adapt more effectively to unfamiliar house layouts.

The Transferability of Rewards vs. Policies

A primary limitation of direct instruction-following policies is their dependency on specific environment geometries. If a target object moves or the room layout changes, the sequence of actions required to reach the goal changes entirely. Conversely, the visual representation of a goal—what a "kitchen table" looks like—remains relatively invariant across different houses. The authors argue that rewards are more transferable than policies; a reward function that recognizes a goal state can be used by an RL agent to plan paths in any new environment.

Methodology: Language-Conditioned Reward Learning (LC-RL)

The authors introduce Language-Conditioned Reward Learning (LC-RL), which utilizes Inverse Reinforcement Learning (IRL) to infer reward functions from expert demonstrations.

Model Architecture

The model employs a multi-modal deep neural network designed to fuse visual and linguistic information into a scalar reward value.

The LC-RL architecture processes panoramic visual input and natural language instructions to output a scalar reward.

As shown in the architecture diagram, the system processes:

  • Vision: A Convolutional Neural Network (CNN) processes a panoramic view of the environment, capturing four cardinal directions (North, South, East, West).
  • Language: A Long Short-Term Memory (LSTM) network processes the natural language command.
  • Fusion: The resulting image and language embeddings are combined via element-wise multiplication (Product) and passed through fully connected layers to produce the final reward signal.

Training and Tractability

To overcome the computational costs of standard IRL, the authors train the model in environments with known dynamics (grid-world abstractions of 3D houses). This allows for exact dynamic programming (Q-iteration) during training, providing stable gradients. Once trained, the reward function is standalone and can be deployed in unknown environments using standard model-free RL algorithms like Deep Q-Networks (DQN).

Experimental Setup and Results

The approach was evaluated using the SUNCG dataset, featuring 1,413 tasks across Navigation (NAV) and Pick-and-Place (PICK) categories. A critical component of the evaluation was the "Test-House" scenario, where models were tested in entirely new house layouts never encountered during training.

A top-down view of a house layout illustrating a navigation task where the agent must identify and reach a specific goal, such as a fruit bowl.

Key Findings

  • Superior Generalization: In unseen house layouts, LC-RL achieved a 36.4% success rate, significantly outperforming the Optimal Policy Cloning baseline, which dropped to 8.5%. This confirms that learning "what" to want is more robust than learning "how" to act.
  • Comparison to Adversarial Methods: LC-RL also outperformed GAIL-Exact. While adversarial methods can mimic expert behavior, their learned discriminators often fail to capture a true reward function that generalizes to new layouts.
  • Visual Grounding: Qualitative analysis showed the reward function successfully "fired" (output high values) only when the agent reached the specific room or object mentioned in the command, demonstrating effective cross-modal grounding.

Conclusion and Future Work

By decoupling goal recognition from action planning, LC-RL provides a more robust framework for instruction following in diverse environments. However, challenges remain, particularly regarding the "exploration bottleneck" at test time; simple random exploration is often insufficient to find goals in complex houses without additional reward shaping. Future research will focus on scaling these methods to continuous state spaces and handling more ambiguous or multi-step instructions.

Summary updated

Abstract

Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.
BibTeX
@inproceedings{fu2019from,
  author = {Justin Fu and Anoop Korattikara and Sergey Levine and Sergio Guadarrama},
  title = {{From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=qTneLhOiHl},
}
Figure