TL;DR
Role-Agent introduces a novel framework that enables a single Large Language Model (LLM) to act concurrently as both an autonomous agent and its own training environment. By coupling future-state prediction with automated failure-mode analysis, this dual-role setup creates a closed-loop, bootstrapped co-evolutionary training process that improves agent performance by over 4% across complex interactive benchmarks without requiring human intervention or separate environment models.
What Problem Does This Paper Address?
While LLM agents excel at complex tasks, training them effectively via Reinforcement Learning (RL) remains a bottleneck. Traditional training environments are typically static, providing sparse and non-specific feedback that limits an agent's ability to generalize.
Conversely, building adaptive synthetic environments—where the environment dynamically adjusts to challenge the agent's specific weaknesses—historically requires complex architectures involving separate environment models, task generators, and scheduling mechanisms. This significantly increases deployment complexity and computational overhead.
The authors of Role-Agent tackle a fundamental question: Can we achieve agent-environment co-evolution by using a single LLM to act as both the agent and the environment?
Methods
Role-Agent splits the single LLM's responsibilities into two synergistic, dual-role components that operate during Agentic Reinforcement Learning (ARL):
- World-In-Agent (WIA): Here, the LLM functions as the agent. During a task rollout, after generating an action, the agent is prompted to explicitly predict the next environment states ($H$ steps into the future). The system measures the alignment between these predictions and the actual ground-truth states using a Longest Matching Subsequence (LMS) metric. This alignment acts as a predictive reward that multiplies the standard task reward. If the agent achieves a goal purely by chance but fails to predict the environment's state changes, its reward is penalized, forcing it to develop reliable, environment-aware reasoning. WIA also incorporates state grouping to calculate fine-grained, state-level advantages.
- Agent-In-World (AIW): Here, the LLM switches roles to act as the environment's curriculum designer. For every failed trajectory, the LLM analyzes the interaction history step-by-step to diagnose the root cause of the failure, generating a structured reflection and a specific failure-mode category (e.g., "entity confusion" or "missing precondition"). The system then uses these insights to retrieve similar tasks from an offline database and dynamically reshapes the training data distribution, ensuring the agent repeatedly practices its historical deficiencies.
Datasets
The framework was evaluated across three distinct types of interactive text-based benchmarks:
- ALFWorld: A household simulator requiring multi-step planning and alignment with physical constraints (e.g., cleaning an object before placing it).
- WebShop: A simulated e-commerce platform containing over 1.18 million real-world products where agents must search and click to fulfill complex user shopping requirements.
- Search-Augmented QA: A suite of single-hop (NQ, TriviaQA, PopQA) and multi-hop (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) question-answering datasets requiring multi-turn web search and information synthesis.
Key Insights and Ideas
- Reliability-Aware Modulation: Instead of simply adding predictive rewards to task rewards, Role-Agent uses multiplication. This prevents the agent from receiving rewards on failed trajectories just for making plausible-sounding state guesses; predictions only matter if they lead to task success.
- Procedural vs. Surface Retrieval: Unlike standard retrieval methods that look for tasks with similar text descriptions, AIW groups and retrieves tasks based on underlying procedural weaknesses. For example, it can link a failed cooking task to a failed cleaning task if both failed due to a "missing precondition" error pattern.
- Mitigating Train-Inference Mismatch: By forcing the agent to internalize world dynamics through state prediction, the gap between training rollouts and inference behavior narrows substantially, leading to lower variance in gradient estimation and highly stable training dynamics.
Key Results
- Consistent Performance Gains: Role-Agent consistently outperformed strong RL baselines (like GRPO and GiGPO) across both Qwen2.5-1.5B and 7B backbones, yielding an average absolute performance gain of over 4%.
- Pronounced Success in Complex Tasks: The framework showed its largest improvements in highly compositional tasks. For instance, on ALFWorld's "Look" and "Pick2" tasks—which demand stable long-term memory and multi-step planning—Role-Agent achieved improvements of +11.0% and +13.6% respectively over GiGPO.
- High Computational Efficiency: Despite the dual-role prompting, the extra computation required for state predictions and failure analysis only introduced a minor 5.2% overhead in total running time per step, proving that co-evolution doesn't require massive computational trade-offs.
Limitations and Future Work
While highly effective, the authors note a few limitations:
- Fairness vs. Capability Trade-off: Keeping the environment LLM identical to the agent backbone ensures a fair baseline comparison, but using a stronger, frozen model (like GPT-4o) for the AIW component could potentially yield even richer environmental feedback.
- Fixed State Grouping Thresholds: The state grouping mechanism relies on a pre-defined similarity threshold (0.9), which may limit its flexibility when generalizing across entirely different task domains.
- Confined to Text: Currently, the framework is restricted to text-based environments. Extending Role-Agent to multi-modal or real-time embodied robotics settings will require integrating vision-language state descriptions or latent-state matching techniques.
Summary updated