Improving Long-Horizon Imitation Through Language Prediction

Donald Joseph Hejna Iii, Pieter Abbeel, Lerrel PintoICLR 2022 · OpenReview

TL;DR

This paper introduces a novel auxiliary loss for imitation learning that trains agents to predict natural language instructions. This approach significantly improves performance on complex, long-horizon planning tasks, especially in low-data regimes, by encouraging agents to learn higher-level, temporally extended representations.

What Problem Does This Paper Address?

Modern deep learning agents struggle with complex, long-horizon planning tasks. This is particularly true when data is scarce, leading to overfitting and compounding errors. While imitation learning can help, it often requires vast amounts of expert demonstrations. Humans, however, excel at planning and generalization by mentally constructing sequences of steps, often guided by language. This paper addresses the challenge of imbuing AI agents with this human-like planning capability, especially in data-limited scenarios, by leveraging language as an auxiliary supervisory signal.

Methods

The core of this work is an auxiliary language prediction loss integrated into a transformer-based imitation learning framework. Instead of solely learning to predict the next action, the agent is also trained to predict high-level natural language instructions that describe how to complete a task. This is achieved by adding a language decoder to a standard transformer encoder-decoder architecture. The encoder processes observations and goals to produce latent representations, which are then used by both the policy (to predict actions) and the language decoder (to predict instructions). This approach allows the agent to learn more abstract, temporally extended representations, crucial for long-horizon planning. The authors emphasize that this instruction prediction is an auxiliary objective, meaning the agent doesn't require explicit instructions at test time; it learns to generate them internally.

Datasets

The proposed method is evaluated on two environments:

  • BabyAI: A procedurally generated, partially observable grid-world environment where agents must complete complex goals specified through language. The authors use a modified version that outputs language instructions.
  • Crafting Environment: A fully observable environment where agents must craft items by collecting and combining resources, with tasks specified by human-generated language instructions.

Key Insights and Ideas

  • Language as a Planning Signal: The paper demonstrates that language instructions, when used as an auxiliary objective, are a powerful tool for improving long-horizon planning in imitation learning. This suggests that language can guide agents towards more abstract and temporally extended reasoning.
  • Data Efficiency: The instruction prediction loss significantly boosts performance in low-data regimes, making imitation learning more feasible and cost-effective. It appears to be more impactful than simply collecting more demonstrations in certain scenarios.
  • Task Complexity Matters: The benefits of instruction prediction are most pronounced in tasks requiring complex reasoning and multi-step planning. For simpler tasks where instructions are easily predictable from goals, the gains are smaller.
  • Scalability of Annotations: The work highlights that collecting language annotations for existing demonstrations can be a more efficient way to improve performance than collecting entirely new demonstrations, especially in data-limited settings.

Key Results

  • In the BabyAI environment, with only 1.25%-5% of the standard demonstration data, the proposed method achieves performance comparable to state-of-the-art approaches that use significantly more data.
  • In the Crafting environment, instruction modeling boosts success rates by about 5% or more with fewer demonstrations, achieving state-of-the-art results.
  • The instruction prediction objective consistently improves or has no negative effect on performance across various dataset sizes and environments.
  • The benefits of instruction modeling are more significant for harder tasks (e.g., 3-step crafting tasks) compared to simpler ones (e.g., 2-step tasks or object localization).

Limitations and Future Work

  • Data Scarcity Impact: The paper notes that with extremely limited data, both the policy and language decoder can overfit, diminishing the benefits of instruction modeling.
  • Task Simplicity: For tasks where instructions are easily predictable from goals alone (e.g., in the ALFRED environment), the instruction prediction loss offers less additional signal and thus less performance improvement.
  • Hierarchical Models: While explored, hierarchical approaches did not consistently outperform the proposed direct instruction prediction method, potentially due to challenges in training accurate high-level policies with limited data.
  • Future Work: The authors suggest further exploring the use of instructions in more complex benchmarks and investigating how task difficulty influences the effectiveness of instruction prediction. They also propose that instructions can become a critical modeling component as task difficulty scales in the future.

Summary updated

Abstract

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents. Difficulties in such settings are exacerbated in low data regimes where over-fitting stifles generalization and compounding errors hurt accuracy. In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. Concretely, we demonstrate that instruction modeling significantly improves performance in planning environments when training with a limited number of demonstrations on the BabyAI and Crafter benchmarks. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans. Our benchmarks and code will be publicly released.
BibTeX
@inproceedings{iii2022improving,
  author = {Donald Joseph Hejna III and Pieter Abbeel and Lerrel Pinto},
  title = {{Improving Long-Horizon Imitation Through Language Prediction}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=1Z3h4rCLvo-},
}
Figure