From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop

Yang Tian, Tianyu Zheng, Xiyuan Yang, Yaxin Du, Chengran Yang, Bo Zhao, Jian Yang, Bryan DaiarXiv 2026 · arXiv

TL;DR

RepoFactory is an automated pipeline that transforms thousands of real-world GitHub repositories into verifiable training environments where AI agents learn to build entire software projects from scratch. By using a novel "Environment Evolution Loop" to optimize task complexity and reliability through low-cost proxy signals, the authors produced a model that outperforms much larger frontier models on long-horizon coding benchmarks.

What Problem Does This Paper Address?

Most current training for coding agents focuses on "short-horizon" tasks: fixing a single bug, writing a unit test, or completing a function. However, real-world software engineering is "long-horizon"—it requires planning across multiple files, managing complex dependencies, and ensuring a whole system works together.

Creating high-quality training data for these long-horizon tasks is notoriously difficult. Existing methods are often "rigid," discarding potentially useful repositories if they don't build perfectly on the first try. Furthermore, improving the "training value" of a task usually requires expensive model rollouts (running the agent to see if it succeeds), which is prohibitively slow and costly when scaled to thousands of environments.

Methods

The authors introduce RepoFactory, a pipeline designed to scale environment synthesis without the high cost of model rollouts. The process follows four main stages:

  1. Search & Filtering: Scouring public code-hosting platforms for testable, isolated Python projects.
  2. Runtime Construction: Building Docker images and ensuring the original test suites pass in a controlled environment.
  3. Task Synthesis: Using a "synthesis agent" to extract API manifests and write natural language specifications that describe the project's functionality.
  4. Environment Evolution Loop: This is the core innovation. Instead of accepting a repository as-is, a "construction agent" iteratively revises the code, tests, and specifications. (See Figure 1)
Figure 1: Overview of RepoFactory. The pipeline searches for repositories, reconstructs their runtimes, and synthesizes task specifications from repository evidence. The Environment Evolution Loop uses low-cost feedback to improve environment reliability and training value.

The loop optimizes a multiplicative objective $Q = A \cdot C \cdot S$:

  • Validity ($A$): Ensures the environment is solvable (the "oracle" passes tests) and verifiable (an empty project fails).
  • Structural Coupling ($C$): Uses static analysis to measure how interdependent the functions are. Higher coupling means the agent can't just solve the project by writing isolated, disconnected functions.
  • Implementation Scope ($S$): Measures the breadth of dependencies and code size, pushing for more substantial reconstruction tasks.

Crucially, this loop relies on "low-cost" feedback—static analysis and test execution—rather than waiting for a full agent rollout.

Datasets

  • RepoFactory Environments: A new collection of 10,159 environments across 17 software domains, including Web APIs, AI/ML, and Systems Infrastructure. (See Figure 2a)
  • SFT Data: 10,947 high-quality trajectories (sequences of actions) collected from 5,577 of these environments.
  • Benchmarks: Evaluation was performed on NL2Repo-Bench (a long-horizon repo generation task), the SWE-bench family (issue resolution), and Terminal-Bench 2.1.
Figure 2: Statistics of RepoFactory environments and SFT data.

Key Insights and Ideas

  • Evolution as a Yield Multiplier: The evolution loop doesn't just improve task quality; it dramatically increases the number of usable environments. By "repairing" repositories that initially failed to build or test, the authors increased the yield of valid environments from 33.8% to 69.4%. (See Figure 6)
  • Proxy Metrics are Effective: You don't need to run a full agent to know if a task is "good" for training. The paper demonstrates that structural coupling and implementation scope are excellent proxies for training value. Optimizing these metrics leads directly to better downstream performance. (See Figure 5)
  • Scaling Without Saturation: Performance on repository-level tasks continues to improve steadily as the number of training environments increases. Even at 5,577 environments, the authors saw no evidence of performance saturation, suggesting that scaling environment diversity is a powerful lever for agentic capabilities. (See Figure 3)
Figure 3: Average accuracy as a function of the number of RepoFactory training environments.

Key Results

The resulting model, RepoHorizen-35B-A3B, showed dramatic improvements over its base model (Qwen3.5-35B-A3B):

  • +22.6 percentage points on NL2Repo-Bench.
  • +8.5 points on average across five diverse benchmarks.
  • The model outperformed much larger open-weight models (like DeepSeek-V4-Pro) and even some frontier closed-source models (like GPT-5.5 and Gemini-3.1 Pro) on the NL2Repo benchmark.
  • The "Environment Evolution" component specifically contributed 5.1 points of the gain on NL2Repo, validating the importance of optimizing task content.
Figure 4: NL2Repo performance across model sizes.

Limitations and Future Work

  • Language Support: RepoFactory currently focuses exclusively on Python. Expanding to other languages like Rust, C++, or Java is a clear next step.
  • Coarse Proxy Metrics: While structural coupling and scope are effective, they are broad. They don't yet distinguish between different types of reasoning or planning capabilities required by a task.
  • Future Directions: The authors suggest incorporating more detailed capability-aware metrics into the evolution loop to support more targeted training for specific agentic skills.

Summary updated

Abstract

Training coding agents for long-horizon tasks requires environments with verifiable feedback. Existing pipelines often follow a linear process that ends with validation, establishing usability but providing little guidance on improving training value. Recursive self-improvement (RSI) often requires model rollouts or model training followed by downstream evaluation, with costs and feedback delays amplified on long-horizon tasks. How can we use timely, reliable proxy signals to improve training value and synthesize environments at scale? We present RepoFactory, an automated pipeline that converts real repositories into verifiable environments where agents build complete projects from scratch given a language specification. Its Environment Evolution Loop optimizes a multiplicative objective combining validity, structural coupling, and implementation scope through proxy feedback. RepoFactory produces 10,159 environments across 17 software domains. Training Qwen3.5-35B-A3B on these environments yields RepoHorizen-35B-A3B, which achieves gains of 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks covering long-horizon coding and agentic tasks in real-world production settings, compared with 17.5 and 6.4 points without environment evolution. These results highlight the training value of RepoFactory environments and the additional gains from the Environment Evolution Loop.
BibTeX
@misc{tian2026from,
  author = {Yang Tian and Tianyu Zheng and Xiyuan Yang and Yaxin Du and Chengran Yang and Bo Zhao and Jian Yang and Bryan Dai},
  title = {{From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop}},
  year = {2026},
  eprint = {2611.21152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21152},
}
Figure