TL;DR
This paper introduces Retrieval-Augmented World Models (RAWM), a novel framework that efficiently integrates external knowledge into LLM-based world models using retrieval-augmented generation and reinforcement learning. RAWM significantly improves prediction accuracy by dynamically retrieving relevant past experiences, enabling LLMs to act as more capable world simulators without requiring extensive retraining.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown immense promise as general-purpose world models, capable of understanding and predicting complex dynamics. However, their performance is often limited by a lack of specific, environment-dependent knowledge. While methods like prompting and fine-tuning can inject this knowledge, they are either computationally expensive, require human expertise for prompt design, or can hurt the LLM's generalizability. This paper addresses the challenge of efficiently equipping LLMs with environment-specific knowledge to enhance their world modeling capabilities without the drawbacks of traditional methods.
Methods
The core of RAWM lies in two key components:
- Retrieval-Augmented Generation (RAG) for World Models: RAWM introduces a memory system that stores pre-collected environmental experiences. An embedding model is used to query this memory for relevant past transitions (state, action, next state) based on the current query (state, action). These retrieved experiences are then provided as in-context examples to the LLM, which acts as the world model, to improve its prediction of the next state. This is inspired by RAG but adapted for the specific task of world modeling.
- Reinforcement Learning (RL) for Embedding Optimization: Instead of directly fine-tuning the LLM, RAWM employs an efficient RL training pipeline to optimize the embedding model. A small MLP head is added to the pre-trained embedding model and trained using Proximal Policy Optimization (PPO). This RL process learns to retrieve better examples, thereby enhancing the world model's prediction performance. This approach is more computationally efficient than fine-tuning the entire LLM.
Datasets
The authors evaluated RAWM on three diverse environments:
- Game24: A mathematical puzzle game where the goal is to reach 24 using four numbers and basic arithmetic operations.
- BlocksWorld: A classic benchmark for evaluating planning capabilities, involving manipulating blocks to achieve a target configuration.
- BabyAI: A suite of partially observable environments based on grid worlds, requiring agents to complete tasks defined by language instructions.
The datasets for these environments were collected using Depth-First Search (for Game24 and BlocksWorld) and a provided bot (for BabyAI), split into memory, query, and test sets.
Key Insights and Ideas
- Retrieval is Crucial for Unseen States: The paper highlights that LLMs struggle to predict states for environments they haven't explicitly encountered. Retrieval of similar past experiences is shown to be vital for accurate predictions in such scenarios.
- RL for Smarter Retrieval: Optimizing the retrieval mechanism through RL is more effective than simply using random in-context examples or relying solely on a pre-trained embedding model. The RL training allows the system to learn which examples are most beneficial for the world model.
- Efficiency without LLM Fine-tuning: RAWM offers a significant advantage by integrating external knowledge without the need for computationally expensive LLM fine-tuning. This preserves the LLM's general capabilities and reduces training overhead.
- Environment-Specific Embeddings: The RL-trained embedding model demonstrates better generalization across different "shots" (number of retrieved examples) and even across different LLM sizes, suggesting it learns environment-specific knowledge rather than just model-specific patterns.
Key Results
RAWM consistently outperformed baseline models across all three environments (Game24, BlocksWorld, and BabyAI).
- Improved Prediction Accuracy: RAWM, especially with the RL-trained embedding, showed significant improvements in prediction accuracy compared to zero-shot (no in-context examples) and random in-context example methods. For instance, in BabyAI with a 1.5B model, RAWM with RL training achieved around 0.48-0.49 accuracy, while random methods were around 0.35-0.40.
- Generalizability: The RL-trained embedding model demonstrated good generalization across different numbers of retrieved examples (K values) and even when transferring from a 1.5B LLM to larger 3B and 7B models.
- Efficiency: The RL training pipeline for the MLP head involves significantly fewer trainable parameters (around 10K) compared to full LLM fine-tuning (1.5B) or LoRA (around 15M), making it much more efficient.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Focus on Next State Prediction: Current RAWM primarily focuses on predicting the next state. Future work will aim to build full-pipeline decision-making systems that leverage RAWM for integrating knowledge.
- Static Dataset: The current RAWM relies on pre-collected datasets. Future work will explore proactive data collection by the system itself to improve performance automatically.
- Text-Based Environments: RAWM is currently designed for text-based environments. Extending it to handle multi-modal environments (e.g., text and images) is a promising future direction.
- Dataset Quality: The performance of RAWM is influenced by the quality of the datasets, similar to LLM training.
- LLM Capabilities: The effectiveness of RAWM is also dependent on the underlying LLM's capabilities. If RL training doesn't yield improvements, considering larger LLMs or memory might be necessary.
Summary updated