UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios

Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, Zixuan Hu, Hongze Mi, Yibo Wang, Naiqiang Tan, Hong Chen, Yi R. Fung, Chun Yuan, Li Shen

Introduction: From Sprints to Marathons

In the rapidly evolving landscape of Large Language Model (LLM) agents, current evaluation frameworks primarily focus on "short-horizon" tasks. Benchmarks like GAIA or SWE-bench typically require fewer than 30 tool calls and a few thousand tokens of context—essentially testing an agent's ability to perform a "sprint." However, real-world scientific discovery and complex software engineering are "marathons" that unfold over long periods in partially observable environments.

To address this evaluation gap, researchers have introduced UltraHorizon, a benchmark designed to stress-test agents in ultra long-horizon scenarios. These tasks demand sustained reasoning, iterative hypothesis testing, and sophisticated memory management over trajectories that can exceed 200,000 tokens and 400 tool calls.

Defining the Ultra Long-Horizon

The authors argue that we are currently "blind" to how agents handle deep time horizons. UltraHorizon pushes the boundaries of existing benchmarks by orders of magnitude in both interaction steps and context volume.

UltraHorizon extends the evaluation space significantly beyond existing benchmarks like GAIA and Tau-Bench in terms of both token count and tool calls.

The Three Environments

UltraHorizon consists of three distinct domains unified by the theme of exploration and discovery:

  1. Mystery Grid: A 10x10 world where agents must infer hidden rules governing movement, score, and energy.
  2. Sequence Exploration: A symbolic domain where agents induce transformation rules within a five-stage pipeline.
  3. Alien Genetics Laboratory: A complex simulation of extraterrestrial inheritance where agents conduct experiments to recover biological mechanisms.

Key Insights: Why Agents Struggle

The benchmark reveals several critical bottlenecks in current agentic AI:

1. The "In-Context Locking" Effect

One of the most striking findings is a form of cognitive inertia. As sequences progress, token entropy declines. Agents become "locked" into early, often incorrect hypotheses and stop exploring new possibilities. This suggests that as the context window fills, the model's ability to pivot or think critically about its own history diminishes.

2. Naive Scaling and Context Overload

Simply increasing the number of allowed interaction steps or the size of the context window does not reliably improve performance. In many cases, agents become overwhelmed by their own history, leading to "context overload" where performance actually degrades as the task lengthens.

3. The Human-AI Gap

While LLMs frequently outperform humans on discrete math or coding puzzles, they lag significantly behind in long-horizon discovery. Humans achieved an average score of 26.52 on UltraHorizon, while the highest-performing model, Gemini-2.5-Pro, reached only 14.33.

Current state-of-the-art LLMs significantly underperform compared to human baselines in long-horizon discovery tasks.

Proposed Solutions: CRNR

To combat these issues, the researchers proposed Context Refresh with Notes Recall (CRNR). Instead of allowing the context window to overflow, CRNR periodically clears the interaction history. To prevent knowledge loss, the agent is required to maintain its own "notes," which it recalls to reconstruct its state after a refresh. This strategy significantly outperformed naive context scaling, suggesting that externalized, structured memory is more effective for long horizons than massive raw context windows.

Conclusion

UltraHorizon highlights a fundamental shift needed in agent development. Success in the real world requires moving beyond foundational reasoning capabilities toward principled memory integration and adaptive strategies that can survive the "marathon" of long-term discovery. Future work will focus on refining these synthetic environments to better capture the nuances of "in-the-wild" scientific and engineering challenges.

Summary updated

BibTeX
@inproceedings{luo2026ultrahorizon,
  author = {Haotian Luo and Huaisong Zhang and Xuelin Zhang and Haoyu Wang and Zeyu Qin and Wenjie Lu and Guozheng Ma and Haiying He and Yingsha Xie and Qiyang Zhou and Zixuan Hu and Hongze Mi and Yibo Wang and Naiqiang Tan and Hong Chen and Yi R. Fung and Chun Yuan and Li Shen},
  title = {{UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=qRNtMWrTvo},
}
Figure