Improving Long-Horizon Imitation through Instruction Prediction

Joey Hejna, Pieter Abbeel, Lerrel Pinto

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans.

1. Introduction

Learning to perform complex, multi-step tasks—such as "find a key, open a door, and move a box"—remains a fundamental challenge in artificial intelligence. Standard Imitation Learning (IL) approaches, where agents mimic expert actions, frequently struggle in two key areas: the low-data regime and the problem of compounding errors. In the former, collecting sufficient expert demonstrations is prohibitively expensive, leading to overfitting. In the latter, small errors in long-horizon tasks lead agents into unfamiliar states, causing total task failure.

While humans utilize internal mental plans or language to organize complex behaviors, most AI agents map observations directly to low-level actions. This work introduces Instruction Prediction, an auxiliary loss that leverages language to help agents learn robust, abstract representations that facilitate better reasoning without requiring instructions at test time.

2. Methodology

The core of our approach is training agents to "translate" their observations into high-level sub-task instructions (e.g., "open the purple door") alongside their primary action prediction task.

2.1 Architecture

The model utilizes a shared encoder—either a Transformer-based sequence model for partially observable tasks or a Vision Transformer (ViT) for fully observable ones. This encoder produces a latent representation $z_t$ from observations $o_t$ and goals $g$. Two heads branch from this latent space: a policy head $\pi_\phi$ that outputs actions $a_t$, and an instruction prediction head $g_\psi$ (a Transformer Decoder) that predicts the sequence of text tokens for the current instruction.

The Instruction Prediction architecture. A Transformer Encoder processes the goal and observation sequence into latents, which are then used for action prediction and instruction decoding. Causal masking ensures the decoder only sees observations up to the current time $t$.

2.2 Loss Function and Causal Masking

The total training loss is a weighted combination of the standard behavior cloning (action classification) loss and a language modeling loss. A critical component of the instruction decoder is the causal masking scheme. The decoder is restricted to seeing observations from the past or present; it cannot access future states. This constraint forces the agent to learn representations that are inherently predictive of the high-level plan. At inference time, the instruction prediction head is discarded, and the agent relies on the high-quality representations learned by the shared encoder.

3. Experimental Setup

We evaluate our method on three challenging benchmarks:

  1. BabyAI: A partially observable grid-world. We focus on the "BossLevel," which requires complex navigation and manipulation.
  2. Crafting Environment: A fully observable environment requiring 1-5 independent steps to craft items, using noisy human-generated instructions.
  3. ALFRED: A visual navigation benchmark used to analyze the correlation between instruction predictability and performance gains.
Sequence of frames showing the agent's performance in the BabyAI environment.

4. Results and Analysis

Our results demonstrate that instruction prediction serves as a powerful "high-level bottleneck," forcing the encoder to extract abstract features that generalize better than raw pixel-level movements.

4.1 Performance and Sample Efficiency

On the BabyAI BossLevel, our method achieved a 70% success rate with only 50,000 demonstrations. This is a significant improvement over the original BabyAI baseline, which required 1 million demonstrations to reach a similar success rate (77%). This suggests that semantic supervision via language is far more sample-efficient than increasing the quantity of raw action data.

4.2 When is Instruction Prediction Most Useful?

Analysis reveals that instruction prediction is most beneficial when instructions cannot be easily inferred from the goal alone. In simple tasks, the auxiliary loss provides minimal gains. However, in tasks requiring complex, non-obvious reasoning, the instruction signal provides a vital training anchor. Furthermore, this approach avoids the "cascading failures" often seen in hierarchical reinforcement learning, as the policy remains flexible and is not strictly bound to a single high-level sub-goal.

5. Conclusion

Instruction prediction significantly improves imitation learning for long-horizon tasks by encouraging the learning of abstract, plan-oriented representations. While the method currently depends on instruction labels during training, its superior sample efficiency and robustness suggest it is a promising direction for complex robotic tasks. Future work will explore applying these techniques to real-world data using automated captioning or multi-modal foundation models.

Summary updated

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans. More details and code can be found at \url{https://github.com/jhejna/instruction-prediction}.
BibTeX
@inproceedings{hejna2023improving,
  author = {Joey Hejna and Pieter Abbeel and Lerrel Pinto},
  title = {{Improving Long-Horizon Imitation through Instruction Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=T68lDkAmAT},
}
Figure