Remember what you did so you know what to do next

Manuel Rafael Ciosici, Alex Hedges, Yash Kankanampati, Justin Andrew Martin, Marjorie Freedman, Ralph M. Weischedel

TL;DR

This paper demonstrates that a moderately sized, self-hosted Large Language Model (LLM), GPT-J (6B parameters), can effectively plan and execute tasks in the ScienceWorld text-based game, significantly outperforming prior reinforcement learning (RL) approaches. Even with limited training data, GPT-J shows remarkable generalization capabilities, challenging previous assumptions about LLMs' suitability for such tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of using LLMs for planning and task execution in complex, simulated environments, specifically the ScienceWorld text game. Previous research (Wang et al., 2022) suggested that LLMs were ill-suited for this domain, performing poorly compared to reinforcement learning (RL) agents. The authors aim to demonstrate that with a more appropriate formulation of the task and by leveraging the LLM's full input capacity, even a moderately sized LLM can achieve state-of-the-art performance. This is crucial for developing more capable AI agents that can understand and act in complex environments without relying on massive, proprietary models.

Methods

The core of the paper's methodology involves fine-tuning the GPT-J LLM (6 billion parameters) on a dataset of ScienceWorld game transcripts. A key innovation is the formulation of the task as a natural language dialog between an agent and the game. This "plain text sequence-modeling format" allows the LLM to leverage its pre-trained knowledge. Crucially, the authors explore filling the LLM's input buffer with as much prior game history (actions and observations) as possible, moving beyond the traditional Markov assumption (conditioning only on the immediate previous step). They also investigate the impact of varying the amount of training data.

Datasets

The primary dataset used is ScienceWorld (SW), a multi-room text game designed to test scientific reasoning abilities. The authors generated training data by creating up to three unique "gold paths" (sequences of actions to achieve goals) for each task variation, resulting in 7359 unique gameplays. They then sampled smaller training sets from this larger dataset, including a "task-balanced" set with only 6.5% of the data (480 games). Evaluation was performed on the full test set of 1819 games.

Key Insights and Ideas

  • LLMs are not inherently poor at complex planning: Contrary to previous findings, this work shows that with the right approach, even a moderately sized LLM can excel in environments previously dominated by RL.
  • Context is king (for LLMs): Filling the LLM's input buffer with extensive game history significantly boosts performance, highlighting the importance of context for LLM reasoning.
  • Generalization from limited data: GPT-J demonstrates impressive generalization capabilities, achieving substantial improvements over baseline RL even when trained on a mere 6.5% of the available data. This suggests LLMs can leverage their pre-trained knowledge effectively.
  • Task-specific performance variation: Averaging performance across all tasks can mask significant differences in how well the LLM performs on specific challenges, indicating that some tasks are inherently more difficult.
  • Preconditions system offers limited score improvement: While a preconditions checker can reduce redundant actions, it doesn't significantly impact the overall game score, likely due to the strict step limit in evaluation.

Key Results

  • 3.5x improvement over DRRN: The "All train" GPT-J model, which uses the full training data and maximizes input history, achieved a 3.5x improvement in score over the best-performing reinforcement learning agent (DRRN).
  • 1.4x improvement with Markov assumption: Even when restricted to the Markov assumption (conditioning only on the previous step), GPT-J outperformed DRRN by a factor of 1.4.
  • 2.2x improvement with 6.5% data: Training on only 6.5% of the data still resulted in a 2.2x improvement over DRRN.
  • Comparable to larger dual-LLM systems: The single 6B GPT-J model achieved performance comparable to a more complex two-stage architecture (SwiftSage) that utilized a much larger LLM (GPT-3.5 Turbo, 29x larger).

Limitations and Future Work

  • LLM unpredictability: The authors caution that LLMs, despite their impressive capabilities, have unforeseen knowledge gaps and unpredictable behavior, making their direct application to real-world robotics or critical systems risky without careful safeguards.
  • Task-specific difficulties: Performance varies significantly across the 30 ScienceWorld tasks, suggesting that some tasks are inherently more challenging for LLMs and require further investigation.
  • Multimodality: The current approach is limited to a text-only environment. Future work should explore integrating LLMs with other modalities like vision and sound to create more versatile agents.
  • External knowledge integration: While the LLM benefits from its pre-training, effectively integrating external knowledge sources (like knowledge graphs) remains an open challenge.
  • Environmental impact: The paper acknowledges the energy consumption of LLMs, particularly during pre-training, and highlights their approach of using a smaller, fine-tuned model to mitigate this.

Summary updated

Abstract

We explore using the 6B parameter GPT-J language model to create a plan for a simulated robot to achieve 30 classes of goals in ScienceWorld, a text game simulator for elementary science experiments and for which previously published empirical work has shown large language models (LLM)s to be a poor fit (Wang et al., 2022). Using the Markov assumption, the LLM outperforms the state-of-the-art based on reinforcement learning by a factor of 1.4. When we fill the LLM’s input buffer with as many prior steps as will fit, improvement rises to 3.3x. Even when training on only 6.5% of the training data, we observe a 2.3x improvement over the state-of-the-art. Our experiments show that performance varies widely across the 30 classes of actions, indicating that averaging over tasks can hide significant performance issues.
BibTeX
@inproceedings{ciosici2023remember,
  author = {Manuel Rafael Ciosici and Alex Hedges and Yash Kankanampati and Justin Andrew Martin and Marjorie Freedman and Ralph M. Weischedel},
  title = {{Remember what you did so you know what to do next}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.104},
}
Figure