Abstract
Understanding hour-long videos presents a "needle in a haystack" challenge for Multimodal Large Language Models (MM-LLMs), where relevant information is often buried under massive amounts of redundant data. We present VideoMiner, a framework that organizes video content into a hierarchical tree structure and utilizes a novel reinforcement learning method, Tree-based Group Relative Policy Optimization (T-GRPO), to iteratively navigate this tree. By making dynamic decisions to accept, continue, or delete nodes, VideoMiner efficiently identifies key frames. Our approach achieves state-of-the-art performance on long-form video benchmarks, including EgoSchema and Video-MME, demonstrating superior scaling as video length increases.
Introduction
The primary bottleneck in long-video understanding is the overwhelming amount of irrelevant "noise" that exceeds the context windows of standard MM-LLMs. Uniform sampling often misses critical temporal details, while simple hierarchical summarization can lose coherence. VideoMiner addresses these issues through two core innovations: a structured video tree for semantic organization and a dynamic policy for adaptive search depth.
Methodology
VideoMiner treats video understanding as a tree-search problem. The process begins with the construction of a Video Tree. The video is first segmented into semantic "events" based on scene changes. A Vision-Language Model (VLM) generates captions for these events, which are then grouped using DBSCAN clustering to form higher-level nodes. This structure preserves temporal flow while significantly reducing complexity.
The navigation of this tree is governed by T-GRPO (Tree-based Group Relative Policy Optimization). Based on the GRPO algorithm, T-GRPO trains a policy model to evaluate each node and take one of three actions:
- ACCEPT: The node contains the necessary information; the search stops.
- CONTINUE: The node is relevant, prompting a deeper search into its children (finer-grained segments).
- DELETE: The node is irrelevant, and its entire branch is discarded.
To balance accuracy and efficiency, we introduce Tree Auxin, a growth parameter inspired by plant biology. This parameter regulates node expansion by weighing the reward of finding an answer against the computational cost of deeper exploration, preventing aimless searching.
Experimental Results
VideoMiner was evaluated across four major long-video benchmarks: EgoSchema, MLVU, Video-MME, and LongVideoBench. It achieved State-of-the-Art (SOTA) results across the board:
- EgoSchema: Scored 66.2, outperforming the previous best (VideoTree) by 6.4 points.
- MLVU: Achieved an average score of 65.1, significantly higher than end-to-end models like Qwen2-VL (60.1).
- Scaling Advantage: A key finding is that the performance gap between VideoMiner and its competitors widened as video length increased, highlighting its efficiency for truly long-form content.
Discussion and Insights
Our analysis revealed that training with T-GRPO encourages the model to develop spontaneous Chain-of-Thought (CoT) reasoning. The model often explains its rationale for accepting or deleting a node before acting, which correlates with a 10% improvement in identifying key frames. Furthermore, we found that clustering based on semantic events is significantly more effective than frame-based clustering for maintaining temporal logic.
While VideoMiner excels in long-form video, it faces a small performance gap on very short clips where the overhead of tree construction is unnecessary. Future work will focus on scaling the architecture with larger base models and optimizing the initial pre-processing costs associated with tree construction.
Summary updated