Improving Long-Horizon Imitation Through Instruction Prediction

Joey Hejna, Pieter Abbeel, Lerrel Pinto

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans.

Introduction

Modern imitation learning (IL) agents frequently struggle with long-horizon tasks where a single error can derail an entire sequence of actions. This challenge is particularly acute in low-data regimes, where agents are prone to two primary failure modes: compounding errors and overfitting. Compounding errors occur when small deviations from the expert trajectory lead the agent into unfamiliar states, while overfitting leads agents to memorize specific trajectories rather than learning the underlying logic of the environment.

While humans often use internal language to structure complex plans—such as "find the key, then open the door"—most IL agents are trained to map observations directly to low-level actions. This research explores whether language can serve as a "scaffold" during training to help agents reason at a higher level of abstraction, even when such language instructions are unavailable during deployment.

Methodology: Instruction Prediction

The core of this approach is an auxiliary instruction prediction loss added to the standard behavior cloning (BC) objective. The architecture utilizes Transformer-based policy networks tailored to the environment: GPT-like sequence models for partially observed settings (e.g., BabyAI) and Vision Transformers (ViT) for fully observed settings (e.g., Crafting).

The Dual-Head Architecture

The model employs a shared observation encoder that feeds into two distinct heads:

  1. Action Head: Predicts the next low-level action (the standard BC objective).
  2. Instruction Decoder Head: Predicts the natural language instruction corresponding to the current segment of the trajectory (e.g., "go to the red door").

Causal Masking

A critical technical component is the use of cross-attention masking. The language decoder is restricted from accessing future observations when predicting an instruction. This constraint forces the model to develop representations that are truly predictive of the high-level plan based solely on the history of observations and the ultimate goal.

Inference

At test time, the instruction prediction head is discarded. The agent operates using only observations and the goal, effectively "thinking" in language internally to guide its actions without requiring human intervention or external guidance during execution.

Experimental Setup

The method was evaluated across three distinct benchmarks to test its versatility and limits:

  • BabyAI (BossLevel): A complex grid-world requiring navigation and object manipulation. The authors modified the environment's oracle to provide step-by-step instructions for training.
  • Crafting Environment: A task requiring agents to collect materials and craft items in sequences of 1 to 5 steps, utilizing real human-generated instructions.
  • ALFRED: A visual navigation and manipulation benchmark used to analyze the relationship between instruction complexity and performance gains.

Key Results and Analysis

Sample Efficiency in BabyAI

The most significant result occurred in the BabyAI "BossLevel." The instruction-augmented model achieved a 70% success rate using only 50,000 demonstrations. In comparison, the original BabyAI baseline required 1 million demonstrations to reach a similar success rate (77%). This suggests that providing high-quality language supervision is significantly more efficient than simply scaling the number of expert demonstrations.

Performance in Crafting

In the Crafting benchmark, instruction modeling (Lang) consistently provided a 5-10% boost in success rates over standard Transformer baselines across various data sizes. Furthermore, the method outperformed Active Temporal Contrast (ATC), a popular self-supervised method, indicating that language provides a more effective signal for planning than generic temporal consistency.

The "Predictability" Threshold

Analysis of the ALFRED benchmark revealed a vital insight: the utility of instruction prediction depends on the information density of the instructions. If a goal (e.g., "Put the watch on the table") makes the intermediate steps obvious, the auxiliary loss provides little benefit. However, in "combinatorially complex" tasks where the path to the goal is not immediately apparent, instruction prediction serves as a powerful representational tool.

Conclusion

By encouraging agents to predict high-level language instructions, we can significantly improve their ability to handle long-horizon tasks in data-constrained environments. This auxiliary loss forces the learning of abstract representations that are more robust to compounding errors and generalize better than standard behavior cloning. While the method requires language annotations during training, the dramatic increase in sample efficiency suggests that language is a highly cost-effective form of supervision for complex robotic and virtual agents.

Summary updated

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans. More details and code can be found at https://github.com/jhejna/instruction-prediction.
BibTeX
@inproceedings{hejna2023improving,
  author = {Joey Hejna and Pieter Abbeel and Lerrel Pinto},
  title = {{Improving Long-Horizon Imitation Through Instruction Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=T68lDkAmAT},
}
Figure