Introduction: From Sprints to Marathons
In the rapidly evolving landscape of Large Language Model (LLM) agents, current evaluation frameworks primarily focus on "short-horizon" tasks. Benchmarks like GAIA or SWE-bench typically require fewer than 30 tool calls and a few thousand tokens of context—essentially testing an agent's ability to perform a "sprint." However, real-world scientific discovery and complex software engineering are "marathons" that unfold over long periods in partially observable environments.
To address this evaluation gap, researchers have introduced UltraHorizon, a benchmark designed to stress-test agents in ultra long-horizon scenarios. These tasks demand sustained reasoning, iterative hypothesis testing, and sophisticated memory management over trajectories that can exceed 200,000 tokens and 400 tool calls.
Defining the Ultra Long-Horizon
The authors argue that we are currently "blind" to how agents handle deep time horizons. UltraHorizon pushes the boundaries of existing benchmarks by orders of magnitude in both interaction steps and context volume.
The Three Environments
UltraHorizon consists of three distinct domains unified by the theme of exploration and discovery:
- Mystery Grid: A 10x10 world where agents must infer hidden rules governing movement, score, and energy.
- Sequence Exploration: A symbolic domain where agents induce transformation rules within a five-stage pipeline.
- Alien Genetics Laboratory: A complex simulation of extraterrestrial inheritance where agents conduct experiments to recover biological mechanisms.
Key Insights: Why Agents Struggle
The benchmark reveals several critical bottlenecks in current agentic AI:
1. The "In-Context Locking" Effect
One of the most striking findings is a form of cognitive inertia. As sequences progress, token entropy declines. Agents become "locked" into early, often incorrect hypotheses and stop exploring new possibilities. This suggests that as the context window fills, the model's ability to pivot or think critically about its own history diminishes.
2. Naive Scaling and Context Overload
Simply increasing the number of allowed interaction steps or the size of the context window does not reliably improve performance. In many cases, agents become overwhelmed by their own history, leading to "context overload" where performance actually degrades as the task lengthens.
3. The Human-AI Gap
While LLMs frequently outperform humans on discrete math or coding puzzles, they lag significantly behind in long-horizon discovery. Humans achieved an average score of 26.52 on UltraHorizon, while the highest-performing model, Gemini-2.5-Pro, reached only 14.33.
Proposed Solutions: CRNR
To combat these issues, the researchers proposed Context Refresh with Notes Recall (CRNR). Instead of allowing the context window to overflow, CRNR periodically clears the interaction history. To prevent knowledge loss, the agent is required to maintain its own "notes," which it recalls to reconstruct its state after a refresh. This strategy significantly outperformed naive context scaling, suggesting that externalized, structured memory is more effective for long horizons than massive raw context windows.
Conclusion
UltraHorizon highlights a fundamental shift needed in agent development. Success in the real world requires moving beyond foundational reasoning capabilities toward principled memory integration and adaptive strategies that can survive the "marathon" of long-term discovery. Future work will focus on refining these synthetic environments to better capture the nuances of "in-the-wild" scientific and engineering challenges.
Summary updated