Abstract
Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.
Introduction
Deep Reinforcement Learning (RL) agents have shown remarkable success in mastering complex motor tasks, yet they remain notoriously "sample hungry." To make language-conditioned RL feasible, researchers often rely on synthetic templates (e.g., "Put the mug on the tray"). While efficient for training, these templates fail to capture the linguistic diversity of real human communication, which includes synonyms, varied phrasing, and frequent typos. This "language gap" prevents agents trained in simulation from effectively interacting with human users.
The SHIFTT Framework
To address this challenge, we introduce SHIFTT (Simulation-to-Human Instruction Following via Transfer from Text), a three-stage methodology designed to bridge the gap between simulated training and human interaction:
- Pretrained Language Integration: We utilize a state-of-the-art language model, such as BERT, which has been pretrained on massive text corpora to capture the nuances of human language.
- RL Training with Frozen Weights: The agent is trained in a 3D environment to perform tasks using synthetic templates. Crucially, the BERT weights are frozen during this stage. This prevents the agent from overfitting to the simplistic synthetic language, thereby preserving its general linguistic knowledge.
- Zero-Shot Human Evaluation: The agent is evaluated on instructions provided by human testers who are not restricted by templates, testing the agent's ability to generalize to natural language without additional training.
The agent's architecture integrates a 3-layer residual convnet for visual processing, the frozen BERT encoder for language, and an LSTM memory core to synthesize information over time.
Experimental Setup
The study utilizes a naturalistic 3D room environment built in Unity. To ensure high visual and linguistic variety, we leverage the ShapeNet dataset, incorporating over 12,000 models across 80 categories.
The diversity of objects within these categories (as seen in Figure 6) requires the agent to learn robust visual representations that map to varied linguistic descriptors.
Key Insights and Results
Our experiments focused on two primary tasks: Reference (identifying a specific object) and Putting (placing one object on another).
Performance on Natural Instructions
The SHIFTT agent demonstrated impressive zero-shot transfer capabilities. In the Reference task, the agent achieved 73% accuracy on natural human instructions, nearly matching the 75% accuracy of human players. In the more complex Putting task, the agent reached 70% accuracy, significantly outperforming a standard word-level baseline which struggled at 36%.
Robustness and Regularization
A key finding was the effectiveness of "Typo Noise"—randomly swapping characters in training strings. This technique not only improved robustness to keyboard errors but also acted as a regularizer, enhancing the agent's ability to handle synonyms. Furthermore, the use of contextual encoders (BERT) proved superior to context-free embeddings, as BERT could resolve phrasal equivalences (e.g., understanding "the container" refers to "the tray" in context).
Conclusion
SHIFTT provides a general recipe for training RL-based systems to interface with human users. By leveraging pretrained text representations and maintaining their integrity during RL training, we achieve substantial zero-shot transfer to natural language. Future work will explore expanding the linguistic scope to include complex conditional logic and interactive dialogue, as well as moving beyond simulated rooms into more complex robotic environments.
Summary updated