Introduction
Reinforcement Learning (RL) has emerged as a powerful paradigm for enhancing the reasoning and tool-use capabilities of Large Language Model (LLM) agents. However, the practical application of RL in agentic workflows is often bottlenecked by the high cost, latency, and lack of scalability associated with real-world environments. While simulated environments offer a potential solution, they typically require extensive manual engineering or rely on LLM-based simulators that are prone to hallucinations and inconsistent feedback.
AutoForge addresses these challenges by introducing a fully automated pipeline for synthesizing complex, executable sandbox environments. By pairing this synthetic data engine with Environment-level Relative Policy Optimization (ERPO)—a novel RL algorithm designed to handle the nuances of multi-turn agentic interactions—AutoForge enables the training of highly capable agents that generalize across diverse domains and languages.
Automated Environment and Task Synthesis
The core of the AutoForge framework is its ability to transform standard tool API documentation into functional, local sandboxes. This process involves two primary stages:
1. Environment Building
Starting from structured API descriptions, an LLM generates a corresponding database schema to represent the environment's state and executable Python functions to represent the tools. This approach creates a fast, deterministic, and reliable sandbox that eliminates the latency of external API calls and the unreliability of purely LLM-driven simulations.
2. Task Generation
To create meaningful training data, AutoForge constructs a dependency graph of the available tools. It samples random walks through this graph to generate valid tool sequences, which are then merged and augmented with "reasoning nodes" to ensure the tasks require complex logical steps. The system executes these sequences to determine verifiable initial and final states, which serve as the ground truth for evaluating agent performance. Finally, these states are used to craft natural language prompts that define the task for the agent.
ERPO: Tailoring RL for Agentic Environments
Standard RL algorithms often struggle with the specific dynamics of agent-user interactions. AutoForge introduces ERPO to mitigate these issues through three key innovations:
- Masking Erroneous Users (MEU): In multi-turn simulations, the "user" is often another LLM. If this simulated user makes a mistake—such as hallucinating a constraint or providing contradictory information—standard RL would penalize the agent for the resulting failure. MEU uses an LLM-as-judge to identify these user errors and mask the affected trajectories, preventing them from destabilizing the training process.
- Environment-Level Advantage Estimation: Training batches in agentic RL often contain tasks from vastly different environments with varying difficulty levels. Global reward normalization can lead to high variance and unstable gradients. ERPO normalizes rewards within each specific environment, ensuring that the advantage signal remains consistent and informative regardless of the task's baseline difficulty.
- Interleaved Thinking: To maintain coherence in long-running tasks, ERPO allows the agent to retain its chain-of-thought reasoning across multiple conversational turns. This prevents the loss of context that often occurs when the reasoning buffer is cleared after every user response.
Experimental Results
The authors evaluated AutoForge by synthesizing 10 virtual environments and 1,078 high-difficulty tasks for training. The resulting model, AutoForge-30B-A3B (built on the Qwen3-Thinking-30B-A3B base), was tested against both open-source and proprietary models.
Performance and Generalization
AutoForge achieved state-of-the-art performance among open-source models under 200B parameters. On benchmarks such as $\tau$-bench (retail and airline) and VitaBench, it demonstrated capabilities comparable to leading closed-source models like GPT-4o-mini and Claude 3.5 Sonnet. For instance, on the $\tau$-bench Retail task, AutoForge scored 73.1, nearly matching Claude 3.5 Sonnet's 73.9.
Crucially, the model exhibited strong out-of-domain generalization. When tested on ACEBench-zh—a Chinese-language benchmark with unseen tools and custom invocation formats—AutoForge showed significant performance improvements over its base model. This suggests that the synthetic training pipeline teaches fundamental reasoning and tool-use principles rather than just overfitting to specific training templates.
Conclusion and Future Work
AutoForge demonstrates that automated environment synthesis is a viable and scalable path toward training sophisticated LLM agents. By addressing the instabilities of multi-turn RL through ERPO and the MEU mechanism, the framework provides a robust foundation for agentic development.
Future research will focus on relaxing the requirement for structured API documentation, allowing for environment generation from more informal descriptions. Additionally, the team plans to explore the scaling laws of synthetic environments and implement turn-level value supervision to further refine the agent's step-by-step decision-making processes.
Summary updated