TL;DR
ReTool-Video introduces a recursive tool-grounding framework and an expansive library of 134 tools to help AI video agents tackle complex, multi-step video reasoning. Instead of forcing abstract goals into rigid, single-step tool calls, it dynamically decomposes high-level intents into executable multimodal tool chains at runtime. This approach achieves state-of-the-art performance on major video understanding benchmarks using a compact 9B parameter model.
What Problem Does This Paper Address?
Complex video question answering (QA) requires more than just looking at a single frame; it demands temporal localization, cross-modal verification, and data aggregation. Current tool-augmented video agents suffer from two main bottlenecks:
- A coarse tool space: Existing agents lack fine-grained operations (like filtering, sorting, or merging) needed for compositional reasoning.
- A flat action space: Standard architectures force an agent to map its high-level intent directly to a single pre-registered tool.
If an agent's intent is abstract (e.g., "check if adjacent clips form one continuous event"), a flat tool-calling mechanism will often fail due to brittle parameters or the lack of an exact tool match.
Methods
The authors address these challenges through two complementary designs:
1. MetaAug-Video Tool Library (MVTL)
Instead of relying on a few coarse tools, the authors built an extensible library of 134 registered tools. It splits capabilities into:
- 26 Base Tools: For general multimodal signal processing (e.g., visual analysis, ASR transcript retrieval, frame inspection).
- 108 Meta Tools: For intermediate-result operations (e.g., filtering, ranking, temporal window merging, mathematical aggregation).
MVTL also supports dual-level video access, allowing the agent to balance efficiency and fidelity by first searching structured data (subtitles, captions) and then verifying with raw modal evidence (timestamped frames).
2. Recursive Tool Grounding (ReTool-Video)
The core innovation is separating intent expression from immediate tool executability. The planner can output primitive actions or abstract video-reasoning intents ($A_{abs}$).
When the planner emits an abstract intent, the system delegates it to a Resolver. The Resolver progressively translates the abstract action into concrete, executable tool chains through parameter repair, tool substitution, or multi-step decomposition at runtime.
3. Reinforcement Learning Optimization
The planner policy is optimized using Group Relative Policy Optimization (GRPO). The training isolates the planner's tokens (masking out the resolver and tool outputs) to focus the learning signal entirely on high-level orchestration, evidence sufficiency judgment, and termination.
Datasets
- Training & Validation: A three-source mixture consisting of Next-QA (causal-temporal reasoning), Video-Bench (comprehensive video LLM tasks), and CFVBench (fine-grained video retrieval-augmented generation).
- Evaluation:
- MVBench: Short-video temporal understanding.
- MLVU: Long-video understanding (ranging from 3 minutes to 2 hours).
- Video-MME (w/o sub): Comprehensive open-domain video QA evaluated in a standard subtitle-free setting.
Key Insights and Ideas
- Abstract Intents as First-Class Citizens: Allowing the planner to express high-level goals rather than forcing immediate tool matching prevents premature execution failures.
- Compositional "Meta" Operations are Crucial: Video reasoning isn't just about perception (seeing a frame); it's about data manipulation. Meta-tools provide the necessary "glue" to filter out distractors and aggregate fragmented clip-level detections.
- Quality Over Quantity: The authors' tool-behavior analysis reveals that longer tool chains do not monotonically improve accuracy. Instead, tool-use success rate and targeted multi-turn reasoning are what drive performance.
Key Results
ReTool-Video (instantiated with Qwen3.5-9B) consistently outperforms much larger open-source and proprietary models:
- MLVU (Long Video): Achieved 81.5% accuracy, significantly outperforming InternVL3.5-30B-A3B (73.0%) and Claude-Sonnet-4.5 (72.8%).
- Video-MME (Subtitle-free): Achieved 76.6% accuracy, surpassing GPT-4o (71.9%) and Gemini-2.5-Flash-Lite (72.7%).
- Ablation Impact: Removing the recursive grounding mechanism caused performance on MLVU to plummet from 74.6% to 63.2% in standard tool-use setups, highlighting the necessity of the Resolver.
Limitations and Future Work
- Inference-Only Resolver: The Resolver currently operates as a non-trainable, inference-only component. Jointly optimizing both the planner and the resolver via RL could yield even more robust tool chains.
- Brittle Deep Recursion: While the system handles abstract actions well, deep recursive steps (capped at a depth of 5 in this paper) run the risk of accumulating schema errors or hitting runtime budget limits.
- Long-tail Failure States: The system remains susceptible to common tool-use failure modes such as missing arguments or empty retrieval results from base perception tools.
Summary updated