Hard questions about long videos usually have to be answered in hops. Take "Why did the woman refuse the drink?" To answer it, a model first has to find the moment she noticed something, then find what she noticed, then link the two. Current video models that search a video step by step write each search request in the same abstract words as the question: "the moment she becomes suspicious." Video does not contain words like "suspicious." It contains a glance held a beat too long, a hand drawn back, a glass seen up close. This proposal claims that much of the failure in multi-hop video reasoning is a vocabulary problem that occurs separately at every hop. The model knows what it is looking for, but it describes it in terms no video segment visibly matches, and it does not say whether the evidence is a detail of appearance or a brief motion.
We propose to train the model to translate each intermediate search need into a concrete observable description, meaning how the clue would look or move on screen, and to send that description to one of two clip indexes. One index stores appearance at high spatial resolution and low frame rate. The other stores motion at high frame rate and low spatial resolution, following the split between slow and fast processing paths that works well for video. The training signal is cheap, in the style of ranking-feedback query rewriting: a rewrite is preferred when the clip it retrieves overlaps the annotated clue interval. The result is a video reasoner whose per-hop searches are retrieved rather than guessed as timestamps. We expect its gains to be concentrated on questions whose wording shares little with what is visible.
Core Claim
The closest system is Video-o3. It handles long-video multi-hop questions by having the model look at a coarse overview, reason, and then decode a call VideoCrop(t_start, t_end, granularity) to inspect a window more closely. It repeats this for up to 8 turns and is trained with reinforcement learning that rewards correct clue intervals. It reports 46.5% on Video-Holmes and 72.1% on MLVU. Its search requests are timestamps the model produces from its own reasoning over a low-resolution overview. There is no step that bridges vocabulary, and no index that a description is matched against.
The diagnostic benchmark Video-Holmes motivates our claim directly. Its questions are written to avoid clue words. Even Gemini-2.5-Pro reaches only 45.0%. About 35% of errors are visual omissions, meaning a fleeting clue was never seen. We claim that a large share of those omissions are vocabulary failures at intermediate hops. If so, rewriting each hop into observable terms and routing it to the right view should reduce omission errors more than reasoning errors, and the gains should grow with the lexical distance between question and clue. If the gains are flat across lexical distance, the claim is wrong.
Research Program
Each video is pre-segmented into short overlapping clips and indexed twice. The appearance view encodes a few frames per clip at full spatial resolution. The motion view encodes dense frames pooled to low spatial resolution, in the manner of the dual-stream projector of SlowFast-LLaVA-1.5. That work processes 128 frames in about 9K tokens by separating "what" from "how." It still samples uniformly and has no search. Both views are embedded into a shared space with a frozen video–text matching encoder. Indexing a clip under two complementary views borrows the idea of MINDER, which represents each text passage by its title, substrings, and synthetic questions. MINDER raised MSMARCO Recall@100 from 57.2 to 78.7 over a single-view generative retriever. We apply the multi-view idea to appearance versus motion rather than to text surfaces.
At each turn, the reasoner, initialized from the Video-o3 checkpoint and trained on its Seeker-173K trajectories, emits three things: a short observable description, a view tag (appearance or motion), and optionally a coarse temporal prior. The description retrieves the top clips from the tagged index. Those clips, at the resolution of that view and under a fixed per-turn token budget, are appended as evidence. We keep Video-o3's answer-time masking of the global overview, which forces answers to rest on retrieved evidence rather than on the overview, because the fake-thinking failure it targets would otherwise hide whether retrieval helped.
The rewriting step is trained with preference pairs, following RaFe. RaFe trains a text query rewriter with DPO, using scores from an off-the-shelf reranker, and reports labelling 30,000 instances in 0.67 hours versus 78 hours with LLM judging. We replace the text reranker, which is blind to video, with a grounding signal. For candidate rewrites $q_1, q_2$ of the same hop, the preferred one has the larger reward
$$r(q) = \max_{c \in \mathrm{top}_k(q)} \mathrm{IoU}(c, g),$$
where $\mathrm{top}_k(q)$ is the set of clips retrieved by $q$ from its tagged view and $g$ is the annotated clue interval for that hop. The reward needs no answer generation, so it stays cheap. Because it rewards only localization, a generic description that retrieves the wrong clip loses. The DPO stage is followed by a short end-to-end stage that reuses Video-o3's trajectory reward, so the reasoner learns when to stop searching.
This design differs from RaFe in three ways. It rewrites at every hop, after earlier evidence has revealed the next entity. Its feedback comes from video grounding rather than text relevance. It also chooses a view. It differs from Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG), which maps middle-layer hidden states into a text retriever to fetch the second hop without decoding. We keep an explicit, inspectable description because the vocabulary bridge is the object of study. A latent variant is tested only as an ablation.
Baselines
The naive reference is the Video-o3 backbone answering from uniformly sampled frames with no search, under the same total visual-token budget. It shows how much any search helps at all.
The strongest existing method is Video-o3, which reports 46.5% on Video-Holmes, 72.1% on MLVU, and 60.7 mIoU on Charades-STA. We will rerun it from the released checkpoint. To keep the comparison fair, we will give it the same additional training compute our method uses, applied as more of its own reinforcement learning on Seeker-173K. It will also use the same 8-turn cap and the same per-turn visual-token budget. Any gain we show must therefore come from rewriting and indexing rather than from extra training. Beating it means a gain on Video-Holmes beyond seed variance (three seeds), with the Video-Holmes error taxonomy showing the reduction comes mainly from visual-omission errors.
The token-efficiency reference is SlowFast-LLaVA-1.5, which reports 71.5% on MLVU and 62.5% on LongVideoBench. We must not lose ground to it on holistic tasks.
For localization we compare to LITA, which combines slow and fast token pooling with 100 relative-time tokens. It reaches 28.6 mIoU on ActivityNet-RTL, a benchmark of conceptual queries such as "working on balance." Its 1%-of-duration time bins are a known limit that retrieval over short clips sidesteps.
Three ablations isolate the key ingredients. The first removes the rewriting step, so the reasoner's raw intermediate thought is used directly as the retrieval query. The second removes the view choice: every query searches both indexes, each at half the per-turn token budget. The third trains the rewriter with a text reranker over clip captions, exactly as in RaFe, instead of the IoU reward.
Why not just…?
Why not just let Video-o3 keep choosing crops itself? This is the main threat. Video-o3's reasoning text may already contain enough to pick the right window, and our gains could be nothing more than extra training. The compute-matched Video-o3 baseline above addresses the extra-training explanation. The lexical-distance analysis described below addresses whether rewriting matters. If Video-o3 with matched compute closes the gap, or if the gap is uniform across lexical distance, the rewriting claim fails.
Why not put a RaFe-style rewriter once at the front? A rewriter that runs before any retrieval cannot name entities that only earlier hops reveal. This is the reason IRCoT interleaves retrieval with reasoning steps and reports recall gains of 11–21 points over one-step retrieval. RaFe's reranker also never sees pixels. We include a single up-front rewrite as a baseline. We expect it to help only on questions with a single clue in Seeker-173K's taxonomy of single-clue versus multi-clue questions.
Why not caption everything and run text multi-hop retrieval? MLVU reports that retrieval helps single-detail tasks but helps little where global perception is needed. Captioners also tend to drop brief gestures, which are exactly Video-Holmes' omission errors. We run IRCoT over dense timestamped captions and speech transcripts as a baseline. If it matches us, then the visual index adds nothing, and that would be a useful negative result.
Why not skip routing and always search both views? At a fixed budget, searching both views halves the resolution of each. That is cheap and might work almost as well. The routing ablation settles it. If both views at half budget match routed retrieval, we drop the view tag and the claim reduces to per-hop rewriting alone.
Proposed Experiments
We evaluate on Video-Holmes for accuracy and error type. We use MLVU's multi-detail tasks, action order and action count, where the benchmark reports sharp degradation. We use ActivityNet-RTL and Charades-STA for localization in mIoU.
The decisive check is a stratified analysis. For each Video-Holmes and ActivityNet-RTL item, we measure lexical distance as the content-word overlap between the question and a reference description of its clue, where reference descriptions are written by annotators for a held-out subset. We then plot the gain over compute-matched Video-o3 against that distance. The hypothesis predicts gains that rise with distance and that fall mostly into the visual-omission category.
A second check is per-hop retrieval recall: the fraction of annotated clue intervals hit at each hop, broken down by hop depth. It tests whether the rewriting step is what fixes the later hops.
A third check measures how often the view choice agrees with a human label of each clue as appearance or motion, and what accuracy is lost when the tag is forced to the wrong view.
Finally, we report wall-clock latency per question and total visual tokens consumed. Video-o3 reports cutting inference time by 32.9–46% over its baseline, so our approach must not give back that saving to count as an improvement.
References
- Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
- Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding
- Multiview Identifiers Enhanced Generative Retrieval
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- LITA: Language Instructed Temporal-Localization Assistant
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding