Abstract
Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 27.7%, 21.1%, and 9.9%, respectively.
Introduction
Reinforcement Learning (RL) has achieved remarkable success in domains with dense, well-defined reward signals. However, real-world applications and complex simulations like Minecraft often present "sparse-reward" environments. In these settings, an agent may execute thousands of actions before encountering a meaningful reward, making exploration extremely difficult.
Traditional solutions rely on intrinsic rewards based on curiosity or novelty. While effective for basic exploration, these methods often lead to "purposeless exploration," where an agent focuses on novel but irrelevant environmental features—a phenomenon known as the "noisy TV" problem. To overcome this, we propose Dreaming with Large Language Models (DLLM), a framework that leverages the high-level reasoning and commonsense priors of Large Language Models (LLMs) to provide purposeful guidance to a model-based RL agent.
The DLLM Framework
DLLM bridges the gap between high-level symbolic reasoning and low-level motor control by integrating LLM-generated "hints" into a world model's internal simulations.
Architecture Components
The system consists of several integrated modules:
- World Model: Built on the Recurrent State-Space Model (RSSM), the world model learns to predict future states, rewards, and natural language descriptions of environmental transitions.
- LLM Hinting Mechanism: At each time step, the agent's observation is converted into a text caption. This caption is processed by an LLM (e.g., GPT-4), which suggests a set of logical subgoals or "hints" (e.g., "craft a wooden pickaxe") based on the current context.
- Goal Embedding and Reward: These text subgoals are embedded into a vector space using SentenceBert. During the world model's internal "dreaming" (rollouts), the agent receives an intrinsic reward if its predicted state transitions align with the suggested subgoals.
- Automatic Descending Mechanism: To ensure the agent continues to progress, we employ Random Network Distillation (RND). This mechanism reduces the intrinsic reward for a specific hint once the agent has consistently mastered it, preventing the agent from stagnating on easy tasks and encouraging it to pursue more complex goals.
Experimental Evaluation
We evaluated DLLM across three challenging environments: HomeGrid, Crafter, and Minecraft. These environments were selected for their sparse rewards and the necessity of long-horizon planning.
Results and Analysis
DLLM demonstrated significant performance improvements over state-of-the-art baselines, including DreamerV3 and Dynalang. In HomeGrid, DLLM improved performance by 27.7% over the strongest baseline. In the Minecraft "Diamond Challenge," it achieved a 9.9% improvement, showing superior data efficiency in the early stages of training.
As shown in the Crafter results, DLLM (particularly when powered by GPT-4) significantly outperforms other methods. The success rate analysis reveals that while most agents can handle simple tasks like "Collect Wood," DLLM excels at "medium-to-hard" tasks such as "Make Stone Pickaxe" or "Collect Iron." These tasks require a specific sequence of prerequisites that standard curiosity-driven agents often fail to discover.
Discussion and Conclusion
The success of DLLM highlights the value of integrating external "commonsense" knowledge into RL agents. By using LLMs to suggest logically sound subgoals, we transform aimless exploration into a purposeful search for achievement. Our results also indicate a direct correlation between the reasoning capabilities of the underlying LLM and the agent's performance, with GPT-4 providing more beneficial exploration hints than GPT-3.5.
Despite its success, DLLM faces challenges regarding LLM stability and the computational cost of API queries. Future research will focus on mitigating the impact of potential LLM hallucinations and exploring the use of smaller, fine-tuned local models to improve efficiency and robustness.
Summary updated