TL;DR
LegoEnv introduces a modular paradigm for building LLM agent training environments using reusable "atoms"—intents, rules, data, and servers. By evolving these shared components when failures occur, it creates a self-improving ecosystem that significantly boosts agent performance across diverse business automation tasks, outperforming existing synthetic pipelines.
What Problem Does This Paper Address?
LLM agents learn best by doing, but they need high-quality environments to practice in. Currently, researchers face a dilemma: manually craft high-quality environments (which doesn't scale) or use LLMs to generate them from scratch (which scales but often results in low-quality, inconsistent, or "broken" tasks).
The core issue with current automated methods is the lack of reuse. If an LLM generates a faulty tool or an inconsistent database for one task, that failure isn't used to improve other similar tasks. There is no shared "memory" or infrastructure. LegoEnv addresses this by breaking environments down into modular, reusable parts, ensuring that a fix in one component improves every environment that uses it.
Methods
The authors propose a "Self-Evolving Agentic Environment" framework based on four types of Atoms:
- Intent Atoms: Specific goals (e.g., "create an invoice").
- Rule Atoms: Business logic or constraints (e.g., "only approve payments over $10,000 if a manager signs off").
- Data Atoms: Real-world public datasets used to seed the environment's state.
- Server Atoms: Tool providers compiled from actual open-source code repositories.
The LegoEnv Pipeline
The process moves through several sophisticated stages:
- Modular Recombination: Atoms are combined into "skeletons" based on structural constraints (e.g., a server must actually possess the tools required by an intent). Figure 2: Atom pools and constraints
- Build & Admit: A builder agent turns the skeleton into a functional environment. It must pass a "correctness check" where a reference solution succeeds, but the initial state does not already satisfy the goal.
- Self-Evolution: If a "teacher" agent fails to solve an admitted environment, a diagnosis agent investigates. It attributes the failure to a specific atom or builder instruction. Fixing that one atom automatically repairs all other environments in the pool that rely on it. Figure 1: Overview of LegoEnv
Datasets
LegoEnv was used to build a corpus of 5,465 verified environments spanning 223 different applications. The authors evaluated their trained agents on three major benchmarks:
- AutomationBench: 600 complex, cross-application business workflows.
- MCP-Mark: 127 tasks involving Model Context Protocol servers (testing transfer to unseen tools).
- BFCL v4 (Berkeley Function Calling Leaderboard): A standard for evaluating tool-use and function-calling capabilities.
Key Insights and Ideas
- The Power of Attribution: In traditional pipelines, a failed training task is often just discarded. In LegoEnv, a failure is a bug report. Because the environment is modular, the system can pinpoint exactly which "Lego brick" is broken.
- Efficiency of Scale: The authors found that 997 individual environment repairs could be reduced to just 33 root causes. This means a small number of architectural fixes can yield a massive increase in solvable training data.
- Real-World Grounding: By using open-source code for servers and real datasets for data, LegoEnv avoids the "uncanny valley" of purely synthetic data, leading to better generalization on real-world tasks.
- Evolutionary Gains: One round of evolution raised the share of solvable environments from 71.6% to 88.3%, proving the loop's effectiveness.
Key Results
LegoEnv demonstrated significant performance lifts across multiple training regimes:
- Supervised Fine-Tuning (SFT): Training Qwen3.6-27B on LegoEnv data doubled its performance on AutomationBench, jumping from 16.0% to 32.0%.
- Reinforcement Learning (RL): Using LegoEnv's programmatic verifiers as rewards in a GRPO (Group Relative Policy Optimization) setup improved a strong Qwen3.8-27B base from 37.6% to 40.3%.
- Competitive Edge: When compared against four other state-of-the-art synthetic environment pipelines (like AWM and EnvFactory) at the same data budget, LegoEnv outperformed them by 37% to 87%. Figure 4: Scaling and Evolution
Limitations and Future Work
- Domain Specificity: The current implementation is focused on English-language business automation. Expanding to other languages or domains (like scientific research or creative coding) remains a future step.
- Agent Dependency: The quality of the evolution loop depends on the capability of the "builder" and "diagnosis" agents. As base models improve, the environment generation will likely become even more robust.
- Convergence: The study covers one major evolution round; it is not yet clear how many rounds are required before the system reaches a point of diminishing returns.
Summary updated