Introduction
Reinforcement learning (RL) agents are typically trained to optimize a reward function. However, defining these functions for complex, vision-based tasks—such as "move the cup from the sink to the kitchen table"—is a significant challenge. While many approaches attempt to map language instructions directly to sequences of actions (policies), these models often fail to generalize to new environments. This paper proposes a shift in perspective: learning a language-conditioned reward function rather than a policy. By grounding language as a goal state rather than a set of moves, agents can adapt more effectively to unfamiliar house layouts.
The Transferability of Rewards vs. Policies
A primary limitation of direct instruction-following policies is their dependency on specific environment geometries. If a target object moves or the room layout changes, the sequence of actions required to reach the goal changes entirely. Conversely, the visual representation of a goal—what a "kitchen table" looks like—remains relatively invariant across different houses. The authors argue that rewards are more transferable than policies; a reward function that recognizes a goal state can be used by an RL agent to plan paths in any new environment.
Methodology: Language-Conditioned Reward Learning (LC-RL)
The authors introduce Language-Conditioned Reward Learning (LC-RL), which utilizes Inverse Reinforcement Learning (IRL) to infer reward functions from expert demonstrations.
Model Architecture
The model employs a multi-modal deep neural network designed to fuse visual and linguistic information into a scalar reward value.
As shown in the architecture diagram, the system processes:
- Vision: A Convolutional Neural Network (CNN) processes a panoramic view of the environment, capturing four cardinal directions (North, South, East, West).
- Language: A Long Short-Term Memory (LSTM) network processes the natural language command.
- Fusion: The resulting image and language embeddings are combined via element-wise multiplication (Product) and passed through fully connected layers to produce the final reward signal.
Training and Tractability
To overcome the computational costs of standard IRL, the authors train the model in environments with known dynamics (grid-world abstractions of 3D houses). This allows for exact dynamic programming (Q-iteration) during training, providing stable gradients. Once trained, the reward function is standalone and can be deployed in unknown environments using standard model-free RL algorithms like Deep Q-Networks (DQN).
Experimental Setup and Results
The approach was evaluated using the SUNCG dataset, featuring 1,413 tasks across Navigation (NAV) and Pick-and-Place (PICK) categories. A critical component of the evaluation was the "Test-House" scenario, where models were tested in entirely new house layouts never encountered during training.
Key Findings
- Superior Generalization: In unseen house layouts, LC-RL achieved a 36.4% success rate, significantly outperforming the Optimal Policy Cloning baseline, which dropped to 8.5%. This confirms that learning "what" to want is more robust than learning "how" to act.
- Comparison to Adversarial Methods: LC-RL also outperformed GAIL-Exact. While adversarial methods can mimic expert behavior, their learned discriminators often fail to capture a true reward function that generalizes to new layouts.
- Visual Grounding: Qualitative analysis showed the reward function successfully "fired" (output high values) only when the agent reached the specific room or object mentioned in the command, demonstrating effective cross-modal grounding.
Conclusion and Future Work
By decoupling goal recognition from action planning, LC-RL provides a more robust framework for instruction following in diverse environments. However, challenges remain, particularly regarding the "exploration bottleneck" at test time; simple random exploration is often insufficient to find goals in complex houses without additional reward shaping. Future research will focus on scaling these methods to continuous state spaces and handling more ambiguous or multi-step instructions.
Summary updated