Abstract
Traditional video reasoning models often struggle with long-form content due to memory constraints and sparse sampling. We present SAGE (Smart Any-horizon aGEnt), a framework that treats video reasoning as an iterative, tool-augmented process. SAGE employs a two-stage workflow—contextual assessment followed by iterative reasoning—to handle videos ranging from seconds to hours. By utilizing a multi-reward Reinforcement Learning (RL) strategy and a synthetic data pipeline, SAGE achieves state-of-the-art performance, outperforming existing models by up to 14.6% on long-video benchmarks.
Introduction
Current state-of-the-art (SOTA) video models typically follow a "DIRECT" paradigm, sampling a fixed number of frames to predict an answer in a single pass. While effective for short clips, this approach fails for "long-horizon" tasks, such as identifying specific details in a feature-length film or complex sporting events. These models often lack the external context or fine-grained temporal resolution required for deep understanding.
SAGE addresses these challenges by mimicking human reasoning patterns. As shown in Figure 1, humans naturally adapt their strategy based on video length: they might watch a short clip in its entirety but "skim" and "search" through long videos using external knowledge and targeted seeking.
Methodology: The SAGE Framework
The core of SAGE is an orchestrator Vision Language Model (VLM) called SAGE-MM, which operates in two distinct stages.
Two-Stage Workflow
- Stage-1 (Context VLM): The model performs an initial pass over global frames and the user query. It determines whether the query can be answered directly or if external tools are required.
- Stage-2 (Iterative Reasoner): If tools are necessary, SAGE enters a multi-turn reasoning loop. It can invoke a suite of six specialized tools: web search, website parsing, speech transcription, event grounding (temporal localization), video part extraction (sub-clip analysis), and visual analysis.
Reinforcement Learning Post-Training
To optimize tool usage and prevent "over-calling," the authors employ Group Relative Policy Optimization (GRPO). This RL strategy uses a multi-reward recipe to score the model on three criteria:
- Format: Adherence to valid JSON output.
- Reasonability: The logical validity of tool calls, evaluated by a teacher model (GPT-4o).
- Accuracy: The correctness of the final predicted answer.
This RL phase proved critical, improving accuracy by 5.7% over standard supervised fine-tuning (SFT) by teaching the model to be more surgical in its tool selection.
Data Generation and Benchmarking
To overcome the high cost of human annotation for long videos, the authors developed a synthetic pipeline using Gemini-2.5-Flash. This pipeline generated 99.1k QnA pairs from over 6,600 YouTube videos, proving to be 100x more cost-effective than traditional methods.
The model was evaluated on SAGE-Bench, a new manually verified benchmark of 1,744 samples with an average duration exceeding 700 seconds. Unlike previous benchmarks focused on multiple-choice questions, SAGE-Bench emphasizes open-ended reasoning in real-world domains like sports and education.
Results and Discussion
SAGE demonstrates superior performance across various horizons:
- Overall Accuracy: SAGE outperformed direct-inference models like Qwen3-VL and Gemini-2.5-Flash by 6.1% on open-ended tasks.
- Long-Video Performance: On videos exceeding 10 minutes, SAGE showed an 8.2% improvement over baselines. When paired with a powerful backend like Gemini-2.5-Flash (SAGE-Flash), this improvement reached 14.6%.
- Efficiency: By predicting timestamps for sub-clips rather than processing entire videos repeatedly, SAGE is approximately 3x faster than existing agentic baselines like VideoMind.
Ablation studies highlighted that speech transcription and video part extraction are the most vital tools, particularly for queries involving verbal information or fine-grained visual details.
Conclusion and Future Work
SAGE represents a significant advancement in "any-horizon" video reasoning. By combining agentic tool-use with RL-driven optimization, it bridges the gap between short-clip understanding and long-form video analysis. Future research will focus on improving temporal grounding accuracy, refining JSON formatting reliability, and expanding the framework to specialized domains such as medical imaging and security surveillance.
Summary updated