Abstract
Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{tabula rasa}$ reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce the Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al., 2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.
Introduction
Reinforcement Learning (RL) has seen remarkable success in environments with well-defined action spaces and dense rewards. However, text-based environments present a unique challenge: the "combinatorial explosion" of possible actions. In complex simulations like ScienceWorld, an agent must interact with its environment using natural language, where every step requires choosing from thousands of potential verb-object combinations (e.g., "connect wire to battery," "open sink," "examine thermometer").
For a $\textit{tabula rasa}$ agent, most of these actions are semantically nonsensical given the current goal. Because rewards are typically sparse—only granted upon the successful completion of a multi-step scientific task—standard RL agents often fail to encounter a reward signal, spending their exploration budget on irrelevant actions. This paper introduces the Language Guided Exploration (LGE) framework to address this bottleneck by leveraging the prior world knowledge embedded in Large Language Models.
The LGE Framework
The LGE framework decouples the decision-making process into two distinct components: a heuristic filter and a specialized learner.
1. The GUIDE
The GUIDE is a BERT-base model fine-tuned using SimCSE, a contrastive learning framework. Its primary role is to act as a semantic filter. Given a specific task description and a list of all possible actions in the current state, the GUIDE ranks these actions by their relevance to the goal. By utilizing pre-trained world knowledge, the GUIDE can identify that "testing conductivity" likely involves a "battery" or "wire" rather than "eating an apple," even before the agent has received any environmental feedback.
2. The EXPLORER
The EXPLORER is a Deep Reinforcement Relevance Network (DRRN). Unlike standard agents that must evaluate the entire action space, the EXPLORER focuses its learning on a restricted subset of actions. Specifically, it primarily samples from the top-$k$ actions recommended by the GUIDE. This transformation reduces a massive search space into a manageable optimization problem. To maintain robustness and account for potential GUIDE errors, the framework employs an epsilon-greedy strategy, allowing the agent to occasionally sample from the full action space.
Experimental Setup
The framework was evaluated on ScienceWorld, a benchmark designed to test scientific reasoning in text-based agents. ScienceWorld includes 30 distinct task types, such as changing states of matter, testing conductivity, and Mendelian genetics, across thousands of variations. The environment is purely text-based, requiring agents to process complex observations and execute multi-step plans.
LGE was compared against several baselines:
- Vanilla RL (DRRN): A standard RL approach without language guidance.
- Behavior Cloning (BC): A supervised learning approach trained on expert trajectories.
- Text Decision Transformer (TDT): A sophisticated architecture that treats RL as a sequence modeling problem.
Results and Key Findings
The LGE framework demonstrated superior performance across the ScienceWorld benchmark, highlighting the efficacy of language-based action pruning.
- Performance Gains: LGE achieved a mean return of 0.23, representing a 35% improvement over the best-performing vanilla RL baseline (0.17). It also outperformed the Text Decision Transformer, suggesting that combining RL with a language filter is more effective than end-to-end sequence modeling for these tasks.
- Pruning Efficiency: The GUIDE proved to be an exceptionally accurate filter. In isolation, it achieved a 0.99 recall at top-50. Out of approximately 2,000 possible actions, the optimal ("gold") action was almost always present in the GUIDE's top 50 suggestions. On average, the rank of the gold action was reduced to just 7.4.
- Generalization: LGE showed strong zero-shot capabilities on unseen variations. Because the GUIDE understands the semantic relationships between objects (e.g., knowing that an "orange" is functionally similar to an "apple" in a nutrition task), it generalizes to new scenarios more effectively than agents that rely solely on environment-specific experience.
Conclusion and Future Work
The LGE framework demonstrates that LLMs do not need to solve a task entirely to be useful; instead, they can serve as powerful filters that align RL exploration with human-like common sense. By pruning the action space, LGE allows agents to learn complex scientific tasks that were previously unreachable for $\textit{tabula rasa}$ systems.
Future research directions include moving toward a "co-training" model where the GUIDE and EXPLORER update simultaneously, and expanding the framework to broader, multi-lingual domains beyond the scientific scope of ScienceWorld.
Summary updated