Abstract
Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like tabula rasa reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce the Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al., 2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.
Introduction
Reinforcement Learning (RL) agents often struggle in text-based environments due to a "combinatorial explosion" of possible actions. In complex environments like ScienceWorld, an agent might face thousands of possible text commands at any given moment (e.g., "connect wire to battery," "eat apple," "look in sink"). When rewards are sparse—meaning the agent only receives feedback after a long sequence of correct steps—randomly discovering the optimal path is computationally infeasible.
While humans leverage common sense to rule out irrelevant actions (e.g., ignoring "eat an apple" when the task is to "test electrical conductivity"), standard RL agents typically start as a tabula rasa and must learn these basic associations from scratch. The Language Guided Exploration (LGE) framework addresses this by injecting "world knowledge" into RL agents to make exploration more efficient, leading to a 35% average performance improvement on the ScienceWorld benchmark.
Methods: The LGE Framework
The LGE framework decouples the decision-making process into two distinct roles: the Guide and the Explorer.
- The Guide: A pre-trained language model (BERT-base) that acts as a semantic filter. It is fine-tuned using contrastive learning (SimCSE) to rank actions based on their relevance to the task description. The Guide learns to embed relevant actions closer to the task description in a shared vector space while pushing irrelevant actions away.
- The Explorer: A standard RL agent, specifically the Deep Reinforcement Relevance Network (DRRN). Instead of navigating the entire action space, the Explorer primarily selects from the top-$k$ (typically 50) actions suggested by the Guide.
To maintain robustness, an $\epsilon$-greedy strategy is employed: the agent usually follows the Guide’s suggestions but occasionally explores the full, unfiltered action space to prevent getting stuck if the Guide's ranking is suboptimal.
Evaluation and Results
The framework was evaluated on ScienceWorld, a complex text-based environment featuring 30 scientific tasks requiring multi-step reasoning. The Guide was trained using approximately 3,400 expert "gold trajectories."
Key Findings
- Semantic Pruning: Task descriptions provide sufficient context to rule out the vast majority of actions. Reducing the action space from thousands to just 50 makes the RL problem significantly more tractable.
- Superior Performance: LGE outperformed the baseline DRRN agent on 20 out of 30 tasks, improving the mean return across all tasks by 35% (from 0.17 to 0.23).
- Generalization: Leveraging a pre-trained language model allows the Guide to generalize to unseen objects. For instance, if it learns "picking up an apple" is relevant for a fruit task, it can infer that "picking up an orange" is also likely relevant.
- High Recall: The Guide achieved a 0.99 recall at $k=50$, ensuring that the correct expert action was almost always present in the filtered list.
LGE also outperformed more complex architectures such as the Text Decision Transformer (TDT) and Behavior Cloning (BC) models, demonstrating that a well-guided, simpler RL agent can be more effective than end-to-end sequence models in these domains.
Limitations and Future Work
Despite its success, LGE has several limitations. It currently depends on expert "gold trajectories" for training the Guide; future iterations could explore self-supervised learning. Additionally, the Guide is currently static during RL training; allowing the Guide and Explorer to co-evolve might yield further performance gains. Finally, while larger models like Macaw were tested, they tended to overfit on the small expert dataset, suggesting that model scale must be carefully balanced with available supervision.
Summary updated