Abstract
Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 41.8%, 21.1%, and 9.9%, respectively.
Introduction
Reinforcement learning (RL) has achieved significant milestones in environments with dense, immediate rewards. However, it often falters in "long-horizon" tasks where an agent must execute a complex sequence of specific actions—such as gathering resources and crafting tools—before receiving any feedback. Standard exploration strategies, like curiosity-driven intrinsic rewards, often lead to "purposeless exploration," where the agent spends time on novel but task-irrelevant states.
To address this, we introduce Dreaming with Large Language Models (DLLM). This framework leverages the high-level reasoning and "common sense" of Large Language Models (LLMs) to provide purposeful guidance to a model-based RL agent. By integrating LLM-generated subgoals into the agent's internal simulations, DLLM narrows the search space and accelerates goal achievement in sparse-reward environments.
Methodology: Dreaming with LLMs
DLLM builds upon the World Model architecture (specifically DreamerV3), which allows an agent to learn a latent dynamics model of its environment and optimize its policy through internal "imagination" or rollouts.
The framework operates through four key mechanisms:
- Multi-modal World Model: The agent learns a latent dynamics model that predicts both visual observations and natural language descriptions of state transitions (e.g., "picked up wood").
- LLM Hinting: Periodically, the agent provides a text description of its current state to an LLM (such as GPT-4). The LLM returns a set of $K$ strategic subgoals or "hints."
- Imaginary Guidance: During policy optimization rollouts in the world model, the agent calculates the similarity between its predicted transition embeddings and the LLM's goal embeddings. Transitions matching the hints receive an intrinsic reward.
- Automatic Reward Decay: To ensure continuous progress, Random Network Distillation (RND) is used to decay the intrinsic reward for subgoals the agent has already mastered, pushing it toward more novel and challenging hints.
Experimental Results
We evaluated DLLM across three increasingly difficult benchmarks: HomeGrid (2D navigation), Crafter (survival game tech tree), and Minecraft (3D resource management). DLLM consistently outperformed strong baselines like Dynalang and DreamerV3.
In HomeGrid, DLLM achieved a 41.8% performance improvement over the strongest baseline. In Crafter, the agent showed a 21.1% improvement, excelling at unlocking medium-to-hard achievements. In the challenging Minecraft Diamond task, DLLM reached a score of 10.0, representing a 9.9% improvement over DreamerV3.
Key Insights
- Purposeful Exploration: DLLM shifts the focus of exploration from novelty to utility. By following LLM hints, the agent avoids wasting time on irrelevant environmental features.
- Scaling with Reasoning: The quality of guidance is directly linked to the LLM's reasoning power. Using GPT-4 resulted in significantly better performance than GPT-3.5.
- Grounding Language: Training the world model to predict language embeddings creates a vital bridge between abstract textual goals and concrete motor actions.
Conclusion
Dreaming with Large Language Models provides a robust bridge between high-level linguistic reasoning and low-level motor control. While challenges remain regarding LLM stability and computational costs, DLLM represents a significant step toward agents that can plan and explore with human-like purpose in open-ended environments.
Summary updated