When a video-language model answers a question like "what did the woman pick up after the phone call?", it has to find the phone call first and only then search the minutes that follow. Today's models do this hopping in text. They write a chain of thought, sometimes call a cropping tool, and rebuild their focus on the video from scratch at every step. Because nothing inside the network remembers where in time it was just looking, two things go wrong. The model loses the thread and misses short clues, and it writes intermediate "observations" that come from language habits rather than from frames it actually attended to.
We propose to give the model an explicit memory of where it is looking in time. This memory is a distribution over frames that is carried from one reasoning step to the next, sharpened or released by a learned gate, and used both to bias the next step's attention and to decide which frames get re-encoded at higher detail. The claim is that this in-model temporal pointer reduces two specific failures: fabricated intermediate observations, and missed clues on long, multi-clue videos. It should do so at a fixed visual token budget and without discrete tool-call turns. The proposal also includes the test that could falsify it: a model that simply writes timestamps in its chain of thought may carry the same pointer for free.
Core Claim
Iterative attention refinement means reusing attention maps from earlier computation instead of recomputing them memorylessly. The idea is well established for depth within a single forward pass. RealFormer adds the previous layer's pre-softmax attention scores to the current layer's, which stabilizes training and yields sparser, more consistent attention. T2MLR passes middle-layer hidden states from one generated token to the next and improves text multi-hop and math reasoning with under 8% inference overhead. Both are text-only, and neither carries an attention state across reasoning steps over a time axis.
Our claim is narrower and testable. Long-video multi-hop reasoning fails largely because the model's focus on the timeline is not persistent across hops. Making that focus a persistent, gated state, and supervising it against annotated clue spans, will lower both the visual-omission errors and the observation hallucinations that dominate current failures. Video-Holmes attributes roughly 35% of failures to visual omission and roughly 60% to reasoning errors. ReWatch-R1 shows that letting a base model "think" more increases ungrounded rationales, so more text reasoning alone is not the fix. We hold the backbone fixed throughout. LVBench reports that backbone reasoning capacity often matters more than long-context engineering, so any gain must come with the backbone held constant.
Mechanism
We build on Qwen2.5-VL-7B, the base model of ReWatch-R1. Call each sentence of the model's reasoning a hop. At the end of hop $k$, we read the attention that a small set of middle-layer heads place on visual tokens, pool it per frame, and normalize it into a frame distribution $a_k$. We keep a temporal state $s_k$ over frames, updated as
$$s_k = g_k\, s_{k-1} + (1-g_k)\, a_k,$$
where $g_k \in [0,1]$ is a scalar gate predicted from the hop's final hidden state. A gate near 1 keeps the previous focus; a gate near 0 replaces it with the new attention. During hop $k+1$, the same heads receive an additive pre-softmax bias $\lambda \log(s_k + \epsilon)$ on each frame's visual tokens. Here $\lambda$ is a learned scale and $\epsilon$ a small floor so that no frame is ever fully excluded. This is RealFormer's residual applied across hops and along the time axis instead of across layers, with a gate.
The gate exists for a specific reason. The research file names over-focusing as the main risk of inherited attention: multiplicative or cumulative schemes such as Progressive Focused Transformer can lock onto early alignments. Multi-hop video questions require abrupt jumps, for example from a clue at second 10 to one at second 500. The gate is how the model releases its old focus.
The state also drives what we call latent zoom. Under a fixed visual token budget, frames where $s_k$ has the least mass are evicted from the KV cache (the stored attention keys and values for earlier tokens). The highest-mass window is re-encoded at a denser frame rate and appended. This is the "scan and zoom" idea of Video-o3, but triggered by the carried state instead of a generated VideoCrop call. With a fixed budget, there is no turn cap set by growing context.
Training has two stages. The first stage uses supervised fine-tuning with an auxiliary loss: a KL divergence pulling $s_k$ toward the annotated clue span for that hop. Clue spans come from the temporal annotations in Video-o3's Seeker-173K trajectories and from the timestamped segments behind the ReWatch dataset. The second stage uses group-relative policy optimization, the RL method behind ReWatch-R1, with answer reward plus a grounding reward. The grounding reward is the overlap between the state's high-mass region and the clue span when the model emits an observation sentence.
This gives a direct hallucination diagnostic. An observation sentence written while $s_k$ is diffuse, or concentrated away from any frame that supports it, is a candidate fabrication. We will check whether this score predicts judge-labelled unsupported observations better than next-token entropy, which is the uncertainty signal DRAGIN uses for text. DRAGIN reads attention only to build a retrieval query; it never feeds attention back into the model.
Relation to Prior Work
Three video systems already connect multi-hop reasoning, video, and hallucination, and all of them do the hopping outside the attention mechanism. ReWatch-R1 synthesizes re-watching chains of thought and rewards verified observations, but its backbone attention is recomputed from scratch at each step. Video-o3 uses tool turns plus task-decoupled attention masks, which hide the low-resolution overview when answering. These masks are fixed per mode rather than a learned, carried state, and the system is capped at 8 turns. Hallucination Mitigation Prompts Long-term Video Understanding wraps frozen modules with question-aware CLIP frame selection and verbalized chain of thought. CLIP selection cannot find a second-hop clue that shares nothing with the question.
Hopper is the closest in mechanism: attention at hop $k$ guides hop $k+1$ over object tracks. However, it is a 6.4M-parameter tracker on synthetic CATER with no language generation. RealFormer and T2MLR supply the method template; Hopper and Video-o3 supply the evidence that hop-to-hop guidance helps on video. The multi-hop text literature, IRCoT and CoRAG, supplies the failure mode we target. In that work, a wrong intermediate hop becomes context that corrupts every later hop.
Baselines
The naive reference point is Qwen2.5-VL-7B with uniform sampling at 192 frames. ReWatch-R1 reports this model at 30.71% average accuracy across five reasoning benchmarks (VCR-Bench, MINERVA, Video Holmes, VideoMathQa, CG-AV-Counting). The second reference is that base model fine-tuned on our exact training mixture with no state, which separates the effect of the data from the effect of the mechanism.
The strongest existing method is Video-o3. It reports 72.1% accuracy on MLVU, 46.5% on Video-Holmes, and 60.7 mIoU on Charades-STA. For context, Gemini-2.5-Pro is reported at 45.0% on Video-Holmes and GPT-4o at 54.5% average on MLVU. ReWatch-R1-7B, at 35.51% on the five-benchmark average, is the strongest process-reward competitor. Video-o3 is the most dangerous comparison because it already does targeted zooming and suppresses "fake thinking."
To make the comparison fair, we will retrain a Video-o3-style tool-calling policy on the same backbone, the same Seeker-173K subset, the same RL step count, and the same total visual token budget per question as our latent zoom. We will also report the published numbers. Beating Video-o3 means exceeding the retrained version at equal visual tokens on Video-Holmes and on MLVU's multi-detail tasks, with a lower observation-hallucination rate. Matching it with fewer tokens or lower latency would be a weaker but still useful result.
We also include the CLIP-selection pipeline from the MovieChat hallucination-mitigation work at the same frame budget. It reports 84.2% global-mode and 62.9% breakpoint-mode accuracy on MovieChat. It tests whether question-to-frame similarity alone suffices.
Three ablations isolate the key ingredient. The first removes carryover by setting $g_k = 0$, so attention is still biased by the current hop's own map. The second is an ungated RealFormer-style residual with $g_k$ fixed near 1, which tests over-focusing. The third removes the clue-span KL and grounding reward, which tests whether the state helps without supervision.
Why not just…?
Why not just have the model write timestamps in its chain of thought? This is the most serious threat. A sentence like "at 3:12 she hangs up" already carries a temporal pointer, and ReWatch-R1 and Video-o3 both produce such text. We train a matched baseline whose hops must name time windows that are then used, through the same latent zoom, to select frames. The only difference from our method is that the pointer lives in text rather than in $s_k$. If this baseline matches ours on accuracy and on hallucination rate, the architectural state is unnecessary, and we will report the work as showing that grounding supervision, not recurrence, is what matters. Our expectation, which the experiment can refute, is that text timestamps fail exactly when the timestamp itself is hallucinated. In that case, a written pointer that is wrong carries forward unchallenged. An attention state, by contrast, is computed from what the model attended to.
Why not just retrofit RealFormer or T2MLR onto the video model? Both refine attention or hidden states across layers or tokens, not across reasoning hops on a frame axis, and neither has been tested on video. A plain additive residual is our ungated ablation. If it matches the gated version, the gate is unnecessary. The file's warning about over-focusing predicts it will instead raise visual-omission errors on questions whose clues are far apart in time. T2MLR also costs 2× to 4× training wall-clock time from its parallel-training approximation, a cost our sentence-level update avoids.
Why not just select frames with CLIP, or sample more frames uniformly? The MLVU Time-ladder shows models that work at 180 seconds collapse toward chance by 600 seconds under uniform sampling. GPT-4o scores 42.9% on MLVU's video needle-in-a-haystack task. CLIP matching on the question cannot locate a second-hop clue, and the CLIP-based pipeline's gains were concentrated on single-timestamp breakpoint queries. Both are run as matched-budget baselines, so this is settled empirically.
Why not just add a stronger process reward, as ReWatch-R1 does? This alternative is partly complementary, so we include the reward in both arms. Running ReWatch-style training with and without the carried state isolates the state's contribution.
Proposed Experiments
The main evaluations are Video-Holmes for clue-chaining, MLVU (including the Time-ladder and the multi-detail tasks Action Ordering and Action Counting), LVBench for hour-long videos, and Charades-STA to check that the state's high-mass region localizes events. Its overlap with ground-truth spans is reported as mIoU against Video-o3's 60.7. Hallucination is measured as the fraction of intermediate observation sentences that a judge model marks unsupported. Judging uses the ReWatch dense timestamped captions as reference, and we hand-audit a sample to check judge agreement.
Four checks are decisive. First, using the Video-Holmes failure taxonomy re-applied to our outputs, does the visual-omission share fall below roughly 35%, and does it fall more than the reasoning-error share? Second, is the accuracy drop from 180 to 600 seconds on the MLVU Time-ladder measurably flatter than the matched no-state model's? Third, an intervention test: we overwrite $s_k$ mid-reasoning with a wrong clue span. If the model's next observation and final answer follow the injected span, the state is causally used; if they do not, the model is ignoring it and any gain comes from training data. Fourth, does the grounding score predict unsupported observations better than token entropy, measured as area under the ROC curve?
The proposal is confirmed if the gated state beats the text-timestamp baseline and the retrained Video-o3 at equal visual tokens on Video-Holmes and on MLVU multi-detail tasks, while lowering the observation-hallucination rate. It is refuted if the text-timestamp baseline closes the gap.
References
- RealFormer: Transformer Likes Residual Attention
- T2MLR: Transformer with Temporal Middle-Layer Recurrence
- Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
- ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis
- LVBench: An Extreme Long Video Understanding Benchmark
- Progressive Focused Transformer for Single Image Super-Resolution
- Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Hallucination Mitigation Prompts Long-term Video Understanding
- Hopper: Multi-hop Transformer for Spatiotemporal Reasoning
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Chain-of-Retrieval Augmented Generation