Abstract
Reinforcement Learning (RL) for Large Language Model (LLM) agents is often hindered by the scarcity of high-quality, environment-specific training tasks. We present CuES (Curiosity-driven and Environment-grounded Synthesis), a novel framework that enables agents to autonomously generate their own training curriculum by exploring new environments. By leveraging intrinsic curiosity and rigorous quality control, CuES synthesizes tasks that are guaranteed to be executable and semantically diverse. Our results show that a 14B parameter model trained with CuES data can outperform models twice its size, achieving significant performance gains on complex tool-use benchmarks like AppWorld and WebShop.
Introduction
The advancement of LLM-based agents relies heavily on their ability to interact with complex environments through tool use. While Reinforcement Learning offers a path to improvement, it requires a structured curriculum of tasks. In many real-world scenarios, such as proprietary APIs or niche software, pre-existing task datasets do not exist—a problem known as task scarcity. Traditional "top-down" approaches using LLMs to brainstorm tasks often result in unexecutable goals, while "bottom-up" random exploration yields low-quality data. CuES bridges this gap by grounding task synthesis in actual environment interactions.
Methodology: The CuES Framework
The CuES framework employs a five-stage pipeline designed to balance open-ended exploration with structured refinement.
- Requirement Confirmation: The system analyzes environment documentation to establish a "concept pool" (entities and actions) and "actionable principles," ensuring the agent understands the sandbox's constraints.
- Curious Exploration: An "Explorer Agent" uses an environment-indexed memory tree to prioritize "unseen" actions, maximizing the discovery of new environment states and avoiding redundant loops.
- Task Abstraction: Raw interaction traces are processed by a "Task Agent" which "lifts" low-level actions into high-level natural language goals and step-by-step guidelines.
- Quality Control: An "Execution Agent" attempts to solve the synthesized task from scratch. Only tasks verified by a "Judge Agent" as successfully re-executable are retained, ensuring 100% executability.
- Goal Rewrite: The system creates a curriculum by varying the difficulty of successful tasks through the manipulation of hints in the goal text.
Experimental Evaluation
We evaluated CuES across three diverse environments: AppWorld (60+ APIs), BFCL v3 (multi-turn function calling), and WebShop (e-commerce navigation). CuES-generated tasks were compared against original human-curated training sets to measure their effectiveness in fine-tuning agentic models.
Results and Discussion
The experimental results demonstrate that autonomous task generation can effectively replace and exceed human-curated data.
Key findings include:
- Significant Performance Gains: On WebShop, a Qwen2.5 14B model trained with CuES achieved 63.55% accuracy, representing a 37.81% improvement over the base model and outperforming the 32B variant (40.17%).
- Efficiency and Scaling: CuES allows smaller open-source models (14B) to approach the performance of much larger closed-source models. On AppWorld, CuES reached 45.24% accuracy, nearing the performance of GPT-4o (48.8%).
- Diversity and Grounding: Visualizations confirm that CuES-generated tasks cover a broader semantic space than human-curated sets. The quality control phase ensures that synthesized tasks represent valid, solvable workflows rather than LLM hallucinations.
Conclusion and Future Work
CuES demonstrates that the bottleneck for agentic RL is not just algorithm design, but the autonomous generation of high-quality training data. While successful, future work will address dependencies on "Judge" LLMs and explore "on-policy" synthesis to target specific agent weaknesses in real-time.
Summary updated