ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding

Xiao Liu, Nayu Liu, Junnan Zhu, Ruirui Chen, Guohui Xiang, Changjian Wang, Kaiwen Wei, Rongzhen Li, Jiang ZhongarXiv 2026 · arXiv

TL;DR

ReTool-Video introduces a recursive tool-grounding framework and an expansive library of 134 tools to help AI video agents tackle complex, multi-step video reasoning. Instead of forcing abstract goals into rigid, single-step tool calls, it dynamically decomposes high-level intents into executable multimodal tool chains at runtime. This approach achieves state-of-the-art performance on major video understanding benchmarks using a compact 9B parameter model.

What Problem Does This Paper Address?

Complex video question answering (QA) requires more than just looking at a single frame; it demands temporal localization, cross-modal verification, and data aggregation. Current tool-augmented video agents suffer from two main bottlenecks:

  1. A coarse tool space: Existing agents lack fine-grained operations (like filtering, sorting, or merging) needed for compositional reasoning.
  2. A flat action space: Standard architectures force an agent to map its high-level intent directly to a single pre-registered tool.

If an agent's intent is abstract (e.g., "check if adjacent clips form one continuous event"), a flat tool-calling mechanism will often fail due to brittle parameters or the lack of an exact tool match.

Figure 1: Motivation of RETOOL-VIDEO. Conventional video tool-using agents force abstract actions (e.g., tem- poral merging or cross-modal verification) into primitive calls, while RETOOL-VIDEO recursively grounds them into executable multimodal tool chains.

Methods

The authors address these challenges through two complementary designs:

1. MetaAug-Video Tool Library (MVTL)

Instead of relying on a few coarse tools, the authors built an extensible library of 134 registered tools. It splits capabilities into:

  • 26 Base Tools: For general multimodal signal processing (e.g., visual analysis, ASR transcript retrieval, frame inspection).
  • 108 Meta Tools: For intermediate-result operations (e.g., filtering, ranking, temporal window merging, mathematical aggregation).

MVTL also supports dual-level video access, allowing the agent to balance efficiency and fidelity by first searching structured data (subtitles, captions) and then verifying with raw modal evidence (timestamped frames).

2. Recursive Tool Grounding (ReTool-Video)

The core innovation is separating intent expression from immediate tool executability. The planner can output primitive actions or abstract video-reasoning intents ($A_{abs}$).

When the planner emits an abstract intent, the system delegates it to a Resolver. The Resolver progressively translates the abstract action into concrete, executable tool chains through parameter repair, tool substitution, or multi-step decomposition at runtime.

Figure 3: Overall framework of RETOOL-VIDEO, where the planner selects primitive or abstract actions, the MetaAug-Video tool library provides base and meta tools, and the resolver recursively grounds abstract actions into executable tool chains.

3. Reinforcement Learning Optimization

The planner policy is optimized using Group Relative Policy Optimization (GRPO). The training isolates the planner's tokens (masking out the resolver and tool outputs) to focus the learning signal entirely on high-level orchestration, evidence sufficiency judgment, and termination.

Datasets

  • Training & Validation: A three-source mixture consisting of Next-QA (causal-temporal reasoning), Video-Bench (comprehensive video LLM tasks), and CFVBench (fine-grained video retrieval-augmented generation).
  • Evaluation:
  • MVBench: Short-video temporal understanding.
  • MLVU: Long-video understanding (ranging from 3 minutes to 2 hours).
  • Video-MME (w/o sub): Comprehensive open-domain video QA evaluated in a standard subtitle-free setting.

Key Insights and Ideas

  • Abstract Intents as First-Class Citizens: Allowing the planner to express high-level goals rather than forcing immediate tool matching prevents premature execution failures.
  • Compositional "Meta" Operations are Crucial: Video reasoning isn't just about perception (seeing a frame); it's about data manipulation. Meta-tools provide the necessary "glue" to filter out distractors and aggregate fragmented clip-level detections.
  • Quality Over Quantity: The authors' tool-behavior analysis reveals that longer tool chains do not monotonically improve accuracy. Instead, tool-use success rate and targeted multi-turn reasoning are what drive performance.

Key Results

ReTool-Video (instantiated with Qwen3.5-9B) consistently outperforms much larger open-source and proprietary models:

  • MLVU (Long Video): Achieved 81.5% accuracy, significantly outperforming InternVL3.5-30B-A3B (73.0%) and Claude-Sonnet-4.5 (72.8%).
  • Video-MME (Subtitle-free): Achieved 76.6% accuracy, surpassing GPT-4o (71.9%) and Gemini-2.5-Flash-Lite (72.7%).
  • Ablation Impact: Removing the recursive grounding mechanism caused performance on MLVU to plummet from 74.6% to 63.2% in standard tool-use setups, highlighting the necessity of the Resolver.

Limitations and Future Work

  • Inference-Only Resolver: The Resolver currently operates as a non-trainable, inference-only component. Jointly optimizing both the planner and the resolver via RL could yield even more robust tool chains.
  • Brittle Deep Recursion: While the system handles abstract actions well, deep recursive steps (capped at a depth of 5 in this paper) run the risk of accumulating schema errors or hitting runtime budget limits.
  • Long-tail Failure States: The system remains susceptible to common tool-use failure modes such as missing arguments or empty retrieval results from base perception tools.

Summary updated

Abstract

Video understanding requires active evidence seeking, motivating tool-augmented video agents for temporal reasoning, cross-modal understanding, and complex question answering. Existing video agents have improved video reasoning with retrieval, memory, frame inspection, and verifier tools, but they still face two limitations: (1) a coarse tool space that lacks fine-grained operations for compositional reasoning; and (2) a flat action space that forces high-level video intents into primitive executable tool calls. In this paper, we address these challenges with two complementary designs. First, we construct a MetaAug-Video Tool Library (MVTL), an extensible tool library with 134 registered tools, including 26 base tools for general multimodal signal processing and 108 meta tools for filtering, aggregation, reranking, formatting, and other intermediate-result operations. MVTL supports dual-level access to both structured video information and raw modal evidence, enabling diverse video reasoning scenarios. Second, we propose ReTool-Video, a recursive tool-using method that grounds high-level video intents into executable tool chains. In ReTool-Video, matched actions are executed directly, while unmatched intents are delegated to a resolver for parameter repair, tool substitution, or decomposition. This allows abstract actions such as temporal merging, cross-modal verification, or repeated-event aggregation to be progressively translated into concrete multimodal operations at runtime. Experiments on MVBench, MLVU, and Video-MME w/o sub. show that ReTool-Video consistently outperforms strong baselines. Further analysis demonstrates that recursive grounding and fine-grained meta tools improve the stability and effectiveness of complex video understanding.
BibTeX
@misc{liu2026retool,
  author = {Xiao Liu and Nayu Liu and Junnan Zhu and Ruirui Chen and Guohui Xiang and Changjian Wang and Kaiwen Wei and Rongzhen Li and Jiang Zhong},
  title = {{ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding}},
  year = {2026},
  eprint = {2605.13228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.13228},
}

Loading related content...

Figure