Abstract
As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective approach to fine-tune LMs with online reinforcement learning (RL) in these environments. We propose Reflect-RL, a two-player system to fine-tune an LM using SFT and online RL, where a frozen reflection model (player) assists the policy model (player). To generate data for the warm-up SFT stage, we use negative example generation to enhance the error-correction ability of the reflection model. Furthermore, we designed single-prompt action enumeration and applied curriculum learning to allow the policy model to learn more efficiently. Empirically, we verify that Reflect-RL outperforms SFT and online RL without reflection. Testing results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B.
Introduction
While Large Language Models (LLMs) excel at static text generation, they often struggle in interactive "trial-and-error" environments—such as navigating computer file systems or executing multi-step robotic tasks. These environments require an agent to adapt its strategy based on environmental feedback. Traditional Supervised Fine-Tuning (SFT) on static datasets often fails to equip models with the necessary error-correction capabilities, as the models are not exposed to the consequences of their own mistakes during training.
To address this, we introduce Reflect-RL, a framework designed to provide smaller language models with an "inner voice" for better decision-making. By decoupling reasoning from acting through a two-player system, Reflect-RL enables models as small as 1.5 billion parameters to outperform significantly larger general-purpose models on complex agentic benchmarks.
The Reflect-RL Framework
The core of Reflect-RL is a two-player architecture that separates the cognitive load of reflection from the execution of actions:
- The Reflection Agent (The "Thinker"): This model is distilled from GPT-4 and trained to analyze the current state and provide a "reflection" on the optimal next step. Once the initial training is complete, this model is frozen to prevent catastrophic forgetting and to provide a stable guidance signal.
- The Policy Agent (The "Doer"): This model receives both the current environmental state and the output from the Reflection Agent. It is trained using online Reinforcement Learning (RL) to select the final action.
Enhancing Learning Efficiency
To optimize the training of these two agents, we implemented several key strategies:
- Negative Example Generation: To ensure the Reflection Agent can identify and correct errors, we intentionally perturbed optimal paths to create "mistakes." We then utilized GPT-4 to generate explanations for why these moves were suboptimal. This process teaches the model to recognize failure modes rather than just mimicking success.
- Single-Prompt Action Enumeration: To eliminate hallucinations and invalid commands, the environment provides a menu of valid actions. The Policy Agent is constrained to output a single token representing its choice, ensuring every action is executable within the environment.
- Curriculum Learning: In environments with sparse rewards, we introduced "milestone" rewards. For instance, in a taxi simulation, the model receives incremental rewards for successfully picking up a passenger, rather than only receiving a reward upon final delivery.
Experimental Setup and Benchmarks
We evaluated Reflect-RL across three distinct interactive environments:
- AutoExplore: A new benchmark requiring agents to navigate complex software repositories using terminal commands (
ls,cd,cat) to answer technical queries. - DangerousTaxi: A high-stakes version of the classic Taxi task where any invalid move (e.g., hitting a wall) immediately terminates the episode, demanding high precision.
- ALFWorld: A standard benchmark for household tasks requiring long-term planning and object manipulation.
Results and Discussion
Reflect-RL demonstrated significant performance gains over both SFT baselines and standard RL approaches without reflection.
Performance vs. Model Size
One of the most striking findings was the efficiency of the framework. A GPT-2 XL (1.56B) model fine-tuned with Reflect-RL achieved a 17% success rate on the AutoExplore task (Depth 2). In contrast, the much larger Mistral 7B model only achieved a 3% success rate. This suggests that specialized reasoning frameworks can allow smaller models to punch far above their weight class in logic-heavy tasks.
Ablation Studies
Ablation tests on the ALFWorld benchmark highlighted the importance of each component. The full Reflect-RL system reached a 74% success rate. Removing the reflection component dropped performance to 66%, while removing the RL fine-tuning entirely (relying only on SFT) resulted in a 0% success rate, illustrating the necessity of online interaction for these tasks.
Success in High-Stakes Environments
In the DangerousTaxi environment, where the margin for error is zero, Reflect-RL achieved a 29% total success rate. Notably, all other baselines—including standard SFT and RL without reflection—failed to complete the task even once (0% success), underscoring the critical role of the "inner voice" in preventing fatal errors.
Conclusion
Reflect-RL demonstrates that the performance of language models in interactive environments is not solely a function of parameter count. By providing a stable, frozen reflection mechanism and training on both successes and failures, we can enable small models to navigate complex, multi-step tasks with high proficiency. This work opens new avenues for deploying efficient, specialized agents in real-world software and robotics applications.
Summary updated