TL;DR
Training terminal agents on longer teacher trajectories doesn't always lead to better performance; in fact, beyond a certain "sweet spot," it can actually hurt reliability while ballooning compute costs. By using a two-stage selective long-horizon refinement strategy—training first on short prefixes and then only on "compatible" long-horizon continuations—researchers achieved higher task success rates with up to 23% less training time.
What Problem Does This Paper Address?
Terminal-based coding agents (like Claude Code or OpenAI’s research agents) operate by executing commands, reading outputs, and iteratively solving software tasks. To train these agents, researchers typically use imitation learning: the agent is supervised to reproduce long "teacher" trajectories generated by more powerful models.
However, a fundamental question has remained unanswered: How much of a trajectory should we actually supervise?
Intuitively, longer horizons should help agents learn complex error recovery and long-range reasoning. But long trajectories are also computationally expensive and increasingly "heterogeneous"—meaning the further you go into a task, the more diverse and noisy the possible histories become. This paper investigates the trade-offs of the supervision horizon (the number of tokens retained for training) and how it shapes agent behavior.
Methods
The authors first characterize the relationship between training horizon and reliability by training a base model (Qwen3-8B) on trajectories from the Nemotron-Terminal-Corpus at various token cutoffs (4K, 8K, 12K, and 16K).
To solve the diminishing returns found in full-horizon training, they propose Selective Long-Horizon Refinement:
- Stage I (Warm-start): Train the model on all available trajectories, but truncated to a short horizon ($H_1$, e.g., 8K tokens).
- Stage II (Scoring): Use this "warm-start" model to calculate the likelihood of the remaining parts of the long trajectories (from $H_1$ to $H_2$).
- Stage III (Selective Refinement): Continue training the model only on the top $\rho\%$ of trajectories that have the highest likelihood (i.e., are most "compatible" with the behavior already learned).
Datasets
- Training:
nvidia/Nemotron-Terminal-Corpus, containing ~60k trajectories generated by DeepSeek-V3.2. - Evaluation:
- Terminal-Bench v1.0 & v2.0: A benchmark for command-line agents.
- OpenThoughts-TBLite: A subset of tasks focused on complex reasoning.
Key Insights and Ideas
- The 12K "Sweet Spot": Reliability (measured by successful task attempts) improves significantly from 4K to 12K but actually drops at 16K. The 12K model solved 29 tasks, while the 16K model solved only 26, despite the latter taking 30% longer to train.
- Horizon Shapes Behavior: The supervision horizon acts as a "persistence dial."
- 4K (Under-persistent): Agents give up too early when they hit an error.
- 12K (Productive persistence): Agents effectively diagnose and recover from errors.
- 16K (Over-persistent): Agents become "obsessive," continuing to act even after a task is solved or looping fruitlessly, leading to much longer inference times.
- The Bias-Complexity Trade-off: The authors formalize the saturation effect using a theoretical bound. Longer horizons reduce temporal supervision bias (the agent learns what to do in the late stages) but increase finite-sample estimation error (the late-stage data is so diverse that the model struggles to generalize from a limited dataset).
- Compatibility Matters More Than Quantity: The success of selective refinement suggests that training on all long-horizon data introduces too much noise. Training only on data that aligns with the agent's current "understanding" (high likelihood) provides a cleaner signal for learning complex behaviors.
Key Results
- Performance Boost: At a 16K horizon, selective refinement (Top-75%) raised successful attempts from 110 to 126 compared to standard full-horizon training.
- Efficiency Gains: Using only 50% of the long-horizon data, the selective method achieved 122 successful attempts while cutting total training time by 23% (39.13h vs 50.81h).
- Generalization: The gains were consistent across benchmarks. On Terminal-Bench v2.0, success increased from 64 to 73; on OpenThoughts-TBLite, it rose from 137 to 155.
- Repeatability: Selective training specifically helped move tasks from "fragile" (solved only once or twice in 8 attempts) to "stable" (solved 6+ times).
Limitations and Future Work
- Imitation vs. Success: The theoretical framework bounds imitation risk (how well the student mimics the teacher), which is a proxy for, but not identical to, actual task success.
- Over-persistence Mechanism: While the paper identifies over-persistence empirically, the theoretical bound doesn't fully explain why agents start looping unnecessarily at very long horizons.
- Teacher Quality: The study assumes a high-quality teacher (DeepSeek-V3.2). It remains to be seen how these horizon effects change if the teacher trajectories themselves are noisier or less optimal.
- Model Scale: The experiments were conducted on 8B-parameter models; exploring whether larger models (e.g., 70B+) are more resistant to the "estimation burden" at long horizons is a promising next step.
Summary updated