Language Models are Few-Shot Butlers

Vincent Micheli, François Fleuret

TL;DR

This paper introduces a two-stage approach to train language models (LMs) to act as agents in text-based environments. By combining imitation learning on a small set of expert demonstrations with reinforcement learning through environment interaction, their "butler" LMs achieve significant performance gains, even with drastically reduced demonstration data.

What Problem Does This Paper Address?

Modern language models excel at many NLP tasks when fine-tuned on task-specific data. The authors propose using these LMs as agents in text-based environments, where understanding and generating language are crucial for interaction. However, a major bottleneck is the acquisition of expert demonstrations. Collecting these demonstrations in text-based environments is incredibly time-consuming, as each demonstration can involve tens of actions over a long horizon to complete multiple sub-goals. This paper aims to overcome this data scarcity by developing a method that can learn effectively from very few demonstrations and then improve through interaction.

Methods

The core contribution is a two-stage training procedure:

  1. Action Modeling (Imitation Learning): Initially, LMs are fine-tuned using a "per-demonstration loss" to imitate a small set of expert demonstrations. This stage is akin to language modeling but focuses on generating actions. The goal is to teach the model the environment's grammar and basic gameplay.
  2. Iterated Action Modeling (Reinforcement Learning): After initial imitation, the model interacts with the environment. Successful trajectories (sequences of actions leading to a goal) are then treated as new, albeit generated, expert demonstrations. The model is further fine-tuned on these successful trajectories using the same action modeling objective. This allows the model to learn from its own successes and improve beyond the initial expert data.

This approach leverages the strengths of both imitation learning (learning from expert examples) and reinforcement learning (learning from self-generated experience).

Datasets

The primary dataset used is from the ALFWorld environment. ALFWorld is a goal-based textual environment designed to mirror embodied AI benchmarks, featuring compositional tasks requiring multiple sub-goals. It provides 3553 training task instances, each with an expert demonstration. The dataset includes 140 in-distribution (seen split) and 134 out-of-distribution (unseen split) evaluation task instances. The paper also mentions using human-annotated goals for robustness evaluation, which are derived from existing ALFWorld annotations.

Key Insights and Ideas

  • Few-shot learning is viable for complex text-based agents: The paper demonstrates that LMs can achieve high performance in interactive environments with surprisingly little expert data, challenging the notion that extensive demonstrations are always necessary.
  • Interaction is key to surpassing limited demonstrations: The iterated action modeling stage is crucial. It allows the model to bootstrap its performance by learning from its own successful interactions, effectively generating more training data and refining its policy.
  • Linguistic priors are beneficial: By using pretrained LMs, the agents benefit from inherent linguistic knowledge, which helps them understand goals and generate valid actions in a natural language-based environment.
  • Robustness to human-like goal formulation: The models show a notable ability to handle goals phrased in natural language that might deviate from the environment's strict generative grammar, highlighting their generalization capabilities.

Key Results

  • Outperforming existing methods: The proposed approach, particularly the GPT*partial model (fine-tuned with iterated action modeling), achieves success rates of 76% (seen split) and 63% (unseen split) using only 1.2% of the expert demonstrations. This represents a 51% absolute improvement over existing methods like BUTLER, which used 14 times more demonstrations.
  • Significant gains with minimal data: Even the GPTpartial model (imitation learning only) with 1.2% of demonstrations performs competitively, outperforming Seq2Seq models trained on much more data.
  • Iterated action modeling's impact: GPT*partial retains 76% and 63% of the performance of GPT (trained on the full dataset) on seen and unseen splits, respectively, while using only 1.2% of the demonstrations.
  • Robustness to human goals: The models solve 57% and 37% of human-annotated, out-of-distribution tasks for GPT and GPT*partial, respectively, demonstrating their ability to generalize to more naturalistic goal formulations.

Limitations and Future Work

  • Performance gap with full data: While impressive, the GPT*partial model still lags behind the GPT model trained on the full dataset, indicating that more data or more sophisticated RL techniques could further improve performance.
  • Environment-specific tuning: The current approach is optimized for the ALFWorld environment. Generalizing this to a wider range of text-based environments might require further adaptation.
  • Future directions: The authors suggest that more sophisticated approaches could be explored, potentially involving further advancements in fine-tuning paradigms or more advanced reinforcement learning techniques. They also note that future work could explore how to achieve great downstream results with even fewer parameter updates, potentially relying more on few-shot priming.

Summary updated

Abstract

Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. Hence, these autoregressive models constitute ideal agents to operate in text-based environments where language understanding and generative capabilities are essential. Nonetheless, collecting expert demonstrations in such environments is a time-consuming endeavour. We introduce a two-stage procedure to learn from a small set of demonstrations and further improve by interacting with an environment. We show that language models fine-tuned with only 1.2% of the expert demonstrations and a simple reinforcement learning algorithm achieve a 51% absolute improvement in success rate over existing methods in the ALFWorld environment.
BibTeX
@inproceedings{micheli2021language,
  author = {Vincent Micheli and François Fleuret},
  title = {{Language Models are Few-Shot Butlers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ZFUIsuOEQO},
}
Figure