Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, Xiangxiang ChuarXiv 2026 · arXiv

TL;DR

Role-Agent introduces a novel framework that enables a single Large Language Model (LLM) to act concurrently as both an autonomous agent and its own training environment. By coupling future-state prediction with automated failure-mode analysis, this dual-role setup creates a closed-loop, bootstrapped co-evolutionary training process that improves agent performance by over 4% across complex interactive benchmarks without requiring human intervention or separate environment models.

What Problem Does This Paper Address?

While LLM agents excel at complex tasks, training them effectively via Reinforcement Learning (RL) remains a bottleneck. Traditional training environments are typically static, providing sparse and non-specific feedback that limits an agent's ability to generalize.

Conversely, building adaptive synthetic environments—where the environment dynamically adjusts to challenge the agent's specific weaknesses—historically requires complex architectures involving separate environment models, task generators, and scheduling mechanisms. This significantly increases deployment complexity and computational overhead.

The authors of Role-Agent tackle a fundamental question: Can we achieve agent-environment co-evolution by using a single LLM to act as both the agent and the environment?

Figure 1: (a): Static environments provide sparse and non-specific feedback that limits the agent’s exploration; (b): Synthetic environments incur high labor and run- time costs; (c): The proposed Role-Agent enables one model to switch roles between agent and environment to achieve bootstrapped co-evolution.

Methods

Role-Agent splits the single LLM's responsibilities into two synergistic, dual-role components that operate during Agentic Reinforcement Learning (ARL):

  1. World-In-Agent (WIA): Here, the LLM functions as the agent. During a task rollout, after generating an action, the agent is prompted to explicitly predict the next environment states ($H$ steps into the future). The system measures the alignment between these predictions and the actual ground-truth states using a Longest Matching Subsequence (LMS) metric. This alignment acts as a predictive reward that multiplies the standard task reward. If the agent achieves a goal purely by chance but fails to predict the environment's state changes, its reward is penalized, forcing it to develop reliable, environment-aware reasoning. WIA also incorporates state grouping to calculate fine-grained, state-level advantages.
  2. Agent-In-World (AIW): Here, the LLM switches roles to act as the environment's curriculum designer. For every failed trajectory, the LLM analyzes the interaction history step-by-step to diagnose the root cause of the failure, generating a structured reflection and a specific failure-mode category (e.g., "entity confusion" or "missing precondition"). The system then uses these insights to retrieve similar tasks from an offline database and dynamically reshapes the training data distribution, ensuring the agent repeatedly practices its historical deficiencies.
Figure 2: Overview of the Role-Agent. A single LLM is leveraged to switch between the roles of agent and environment. As an agent, it is prompted to predict states for the next H steps; the alignment between these predictions and ground-truth states serves as a reward signal to compute trajectory-level and state-level advantages. As the environment, it analyzes failure modes from failed trajectories and reshapes the data distribution by retrieving tasks with similar modes. This closed-loop process enables bootstrapped agent-environment co-evolution.

Datasets

The framework was evaluated across three distinct types of interactive text-based benchmarks:

  • ALFWorld: A household simulator requiring multi-step planning and alignment with physical constraints (e.g., cleaning an object before placing it).
  • WebShop: A simulated e-commerce platform containing over 1.18 million real-world products where agents must search and click to fulfill complex user shopping requirements.
  • Search-Augmented QA: A suite of single-hop (NQ, TriviaQA, PopQA) and multi-hop (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) question-answering datasets requiring multi-turn web search and information synthesis.

Key Insights and Ideas

  • Reliability-Aware Modulation: Instead of simply adding predictive rewards to task rewards, Role-Agent uses multiplication. This prevents the agent from receiving rewards on failed trajectories just for making plausible-sounding state guesses; predictions only matter if they lead to task success.
  • Procedural vs. Surface Retrieval: Unlike standard retrieval methods that look for tasks with similar text descriptions, AIW groups and retrieves tasks based on underlying procedural weaknesses. For example, it can link a failed cooking task to a failed cleaning task if both failed due to a "missing precondition" error pattern.
  • Mitigating Train-Inference Mismatch: By forcing the agent to internalize world dynamics through state prediction, the gap between training rollouts and inference behavior narrows substantially, leading to lower variance in gradient estimation and highly stable training dynamics.

Key Results

  • Consistent Performance Gains: Role-Agent consistently outperformed strong RL baselines (like GRPO and GiGPO) across both Qwen2.5-1.5B and 7B backbones, yielding an average absolute performance gain of over 4%.
  • Pronounced Success in Complex Tasks: The framework showed its largest improvements in highly compositional tasks. For instance, on ALFWorld's "Look" and "Pick2" tasks—which demand stable long-term memory and multi-step planning—Role-Agent achieved improvements of +11.0% and +13.6% respectively over GiGPO.
  • High Computational Efficiency: Despite the dual-role prompting, the extra computation required for state predictions and failure analysis only introduced a minor 5.2% overhead in total running time per step, proving that co-evolution doesn't require massive computational trade-offs.
Figure 3: Running dynamics on ALFWorld. (left): suc- cess rate on the validation set; (right): the averaged difference between training and inference rollouts.

Limitations and Future Work

While highly effective, the authors note a few limitations:

  1. Fairness vs. Capability Trade-off: Keeping the environment LLM identical to the agent backbone ensures a fair baseline comparison, but using a stronger, frozen model (like GPT-4o) for the AIW component could potentially yield even richer environmental feedback.
  2. Fixed State Grouping Thresholds: The state grouping mechanism relies on a pre-defined similarity threshold (0.9), which may limit its flexibility when generalizing across entirely different task domains.
  3. Confined to Text: Currently, the framework is restricted to text-based environments. Extending Role-Agent to multi-modal or real-time embodied robotics settings will require integrating vision-language state descriptions or latent-state matching techniques.

Summary updated

Abstract

Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization. To address these limitations, this paper introduces Role-Agent, \textcolor{black}{a framework} that harnesses a single LLM to function concurrently as both the agent and the environment, enabling a bootstrapped co-evolution. Role-Agent comprises two synergistic components: World-In-Agent (WIA) and Agent-In-World (AIW). In WIA, the LLM acts as the agent and predicts future states after each action; the alignment between predicted and actual states is then used as a process reward, encouraging environment-aware reasoning. In AIW, the LLM analyzes failure modes from failed trajectories and retrieves tasks with similar failure patterns, thereby reshaping the training data distribution for targeted practice. Experiments on multiple benchmarks show that Role-Agent consistently improves performance, yielding an average gain of over 4\% over strong baselines.
BibTeX
@misc{wang2026role,
  author = {Xucong Wang and Ziyu Ma and Shidong Yang and Tongwen Huang and Pengkun Wang and Yong Wang and Xiangxiang Chu},
  title = {{Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution}},
  year = {2026},
  eprint = {2606.10917},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.10917},
}

Loading related content...

Figure