Reflect-RL: Two-Player Online RL Fine-Tuning for LMs

Runlong Zhou, Simon S. Du, Beibin Li

Abstract

As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective approach to fine-tune LMs with online reinforcement learning (RL) in these environments. We propose Reflect-RL, a two-player system to fine-tune an LM using SFT and online RL, where a frozen reflection model (player) assists the policy model (player). To generate data for the warm-up SFT stage, we use negative example generation to enhance the error-correction ability of the reflection model. Furthermore, we designed single-prompt action enumeration and applied curriculum learning to allow the policy model to learn more efficiently. Empirically, we verify that Reflect-RL outperforms SFT and online RL without reflection. Testing results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B.

The Challenge of Interactive Agents

While Large Language Models (LLMs) excel at reasoning and coding in static contexts, they often struggle as autonomous agents in interactive environments—such as navigating a file system or managing a robot in a house. Standard Supervised Fine-Tuning (SFT) on static datasets fails to capture the "trial-and-error" nature of real-world interactions. Furthermore, LLMs frequently fail to correct their own mistakes without external feedback, and sparse rewards in complex tasks make it difficult for standard RL to identify which specific actions led to success.

The Reflect-RL Framework

Reflect-RL is a novel "two-player" system designed to bridge the gap between reasoning and acting. The architecture consists of a Reflection Agent and a Policy Agent. Before taking an action, the Reflection Agent (distilled from GPT-4) analyzes the current state and previous steps to suggest a plan. The Policy Agent then takes this reflection as input to decide the next move.

The Reflect-RL architecture showing the interaction between the Reflection Agent, Policy Agent, and the environment.

To make the system robust, the authors implemented several key strategies:

  • Negative Example Generation: To prevent the model from becoming a "yes-man" that only knows how to follow expert paths, the authors intentionally perturbed optimal trajectories to create "wrong" moves. They used GPT-4 to explain why those moves were bad and how to fix them, teaching the model to handle failure.
  • Single-Prompt Action Enumeration: To prevent command hallucinations, the environment lists all valid actions in the prompt. The model is then restricted to outputting a single token representing its choice.
  • Curriculum Learning: To solve the sparse reward problem, "milestone" rewards were introduced to guide the model through long-horizon tasks, such as giving a small reward for picking up a passenger before the final drop-off.

Experimental Evaluation

The researchers evaluated Reflect-RL on three distinct environments:

  1. AutoExplore: A new benchmark where an agent navigates a complex file system to find specific information.
  2. DangerousTaxi: A high-stakes version of the classic Taxi task where any invalid move results in an immediate game over.
  3. ALFWorld: A standard benchmark for household tasks requiring multi-step planning in a text-based environment.

Performance and Results

The performance gains were substantial across all benchmarks. Notably, a relatively small GPT-2 XL (1.56B) model fine-tuned with Reflect-RL was able to outperform much larger 7B parameter models.

Performance comparison of Reflect-RL against open-source models and various ablation baselines.

Key results include:

  • AutoExplore: Reflect-RL achieved a 36% success rate on Depth 1 tasks, outperforming Mistral 7B (34%) and Llama-2 7B (2%).
  • DangerousTaxi: Reflect-RL achieved a 29% full completion rate. Remarkably, all other baselines, including Mistral 7B and standard RL-tuned GPT-2, scored 0%.
  • ALFWorld: Reflect-RL reached a 74% success rate, whereas the base Mistral 7B and Llama-2 models failed to complete a single task.

Ablation studies confirmed that removing "negative examples" from the training data dropped the success rate on AutoExplore from 36% to 33%, proving that learning from mistakes is critical for resilience.

Discussion and Future Work

Reflect-RL demonstrates that for specialized interactive tasks, a small, well-tuned model with a dedicated reflection mechanism can be more effective than a massive, general-purpose model. By decoupling reasoning (reflection) and action (policy), the authors simplified the RL training process.

However, limitations remain. The reflection model was frozen during the RL phase; future work could explore "co-training" both models simultaneously. Additionally, while this work focused on text-based representations, extending Reflect-RL to handle raw visual or auditory inputs in embodied AI settings remains a promising direction for future research.

Summary updated

Abstract

As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective approach to fine-tune LMs with online reinforcement learning (RL) in these environments. We propose Reflect-RL, a two-player system to fine-tune an LM using SFT and online RL, where a frozen reflection model (player) assists the policy model (player). To generate data for the warm-up SFT stage, we use negative example generation to enhance the error-correction ability of the reflection model. Furthermore, we designed single-prompt action enumeration and applied curriculum learning to allow the policy model to learn more efficiently. Empirically, we verify that Reflect-RL outperforms SFT and online RL without reflection. Testing results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B. The benchmarks, dataset, and code involved in this work are publicly available: https://github.com/zhourunlong/Reflect-RL.
BibTeX
@inproceedings{zhou2024reflect,
  author = {Runlong Zhou and Simon S. Du and Beibin Li},
  title = {{Reflect-RL: Two-Player Online RL Fine-Tuning for LMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Wrxdj2kBr0},
}
Figure