CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL

Shinji Mai, Yunpeng Zhai, Ziqian Chen, Cheng Chen, Anni Zou, Shuchang Tao, Zhaoyang Liu, Bolin DingarXiv 2025 · DOI · arXiv

Abstract

Reinforcement Learning (RL) for Large Language Model (LLM) agents is often hindered by the scarcity of high-quality, environment-specific training tasks. We present CuES (Curiosity-driven and Environment-grounded Synthesis), a novel framework that enables agents to autonomously generate their own training curriculum by exploring new environments. By leveraging intrinsic curiosity and rigorous quality control, CuES synthesizes tasks that are guaranteed to be executable and semantically diverse. Our results show that a 14B parameter model trained with CuES data can outperform models twice its size, achieving significant performance gains on complex tool-use benchmarks like AppWorld and WebShop.

Introduction

The advancement of LLM-based agents relies heavily on their ability to interact with complex environments through tool use. While Reinforcement Learning offers a path to improvement, it requires a structured curriculum of tasks. In many real-world scenarios, such as proprietary APIs or niche software, pre-existing task datasets do not exist—a problem known as task scarcity. Traditional "top-down" approaches using LLMs to brainstorm tasks often result in unexecutable goals, while "bottom-up" random exploration yields low-quality data. CuES bridges this gap by grounding task synthesis in actual environment interactions.

Methodology: The CuES Framework

The CuES framework employs a five-stage pipeline designed to balance open-ended exploration with structured refinement.

The five-stage CuES pipeline for environment-grounded task synthesis.
  1. Requirement Confirmation: The system analyzes environment documentation to establish a "concept pool" (entities and actions) and "actionable principles," ensuring the agent understands the sandbox's constraints.
  2. Curious Exploration: An "Explorer Agent" uses an environment-indexed memory tree to prioritize "unseen" actions, maximizing the discovery of new environment states and avoiding redundant loops.
  3. Task Abstraction: Raw interaction traces are processed by a "Task Agent" which "lifts" low-level actions into high-level natural language goals and step-by-step guidelines.
  4. Quality Control: An "Execution Agent" attempts to solve the synthesized task from scratch. Only tasks verified by a "Judge Agent" as successfully re-executable are retained, ensuring 100% executability.
  5. Goal Rewrite: The system creates a curriculum by varying the difficulty of successful tasks through the manipulation of hints in the goal text.

Experimental Evaluation

We evaluated CuES across three diverse environments: AppWorld (60+ APIs), BFCL v3 (multi-turn function calling), and WebShop (e-commerce navigation). CuES-generated tasks were compared against original human-curated training sets to measure their effectiveness in fine-tuning agentic models.

Results and Discussion

The experimental results demonstrate that autonomous task generation can effectively replace and exceed human-curated data.

Performance comparison of CuES-trained models against larger baselines across AppWorld, WebShop, and BFCLv3.

Key findings include:

  • Significant Performance Gains: On WebShop, a Qwen2.5 14B model trained with CuES achieved 63.55% accuracy, representing a 37.81% improvement over the base model and outperforming the 32B variant (40.17%).
  • Efficiency and Scaling: CuES allows smaller open-source models (14B) to approach the performance of much larger closed-source models. On AppWorld, CuES reached 45.24% accuracy, nearing the performance of GPT-4o (48.8%).
  • Diversity and Grounding: Visualizations confirm that CuES-generated tasks cover a broader semantic space than human-curated sets. The quality control phase ensures that synthesized tasks represent valid, solvable workflows rather than LLM hallucinations.

Conclusion and Future Work

CuES demonstrates that the bottleneck for agentic RL is not just algorithm design, but the autonomous generation of high-quality training data. While successful, future work will address dependencies on "Judge" LLMs and explore "on-policy" synthesis to target specific agent weaknesses in real-time.

Summary updated

Abstract

Large language model based agents are increasingly deployed in complex, tool augmented environments. While reinforcement learning provides a principled mechanism for such agents to improve through interaction, its effectiveness critically depends on the availability of structured training tasks. In many realistic settings, however, no such tasks exist a challenge we term task scarcity, which has become a key bottleneck for scaling agentic RL. Existing approaches typically assume predefined task collections, an assumption that fails in novel environments where tool semantics and affordances are initially unknown. To address this limitation, we formalize the problem of Task Generation for Agentic RL, where an agent must learn within a given environment that lacks predefined tasks. We propose CuES, a Curiosity driven and Environment grounded Synthesis framework that autonomously generates diverse, executable, and meaningful tasks directly from the environment structure and affordances, without relying on handcrafted seeds or external corpora. CuES drives exploration through intrinsic curiosity, abstracts interaction patterns into reusable task schemas, and refines them through lightweight top down guidance and memory based quality control. Across three representative environments, AppWorld, BFCL, and WebShop, CuES produces task distributions that match or surpass manually curated datasets in both diversity and executability, yielding substantial downstream policy improvements. These results demonstrate that curiosity driven, environment grounded task generation provides a scalable foundation for agents that not only learn how to act, but also learn what to learn. The code is available at https://github.com/modelscope/AgentEvolver/tree/main/research/CuES.
BibTeX
@misc{mai2025cues,
  author = {Shinji Mai and Yunpeng Zhai and Ziqian Chen and Cheng Chen and Anni Zou and Shuchang Tao and Zhaoyang Liu and Bolin Ding},
  title = {{CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agentic RL}},
  year = {2025},
  eprint = {2512.01311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.01311},
}
Figure