Video language models often describe things a video does not show. They say an event came before another when it came after. They report a scene change that never happened. They name an action because the background usually goes with it, for example "skiing" whenever there is snow. In parallel, video is extremely repetitive: most pixels in adjacent frames are the same, which is why token-merging methods can drop half the visual input at little cost. This proposal borrows one idea from decentralized control and uses that repetition to reduce the false descriptions. The idea is the common information approach: a method for coordinating agents that each see only part of the world. Each agent's knowledge is split into what everyone knows (common information) and what only that agent knows (private information). A fictitious coordinator that sees only the common part then sends each agent a prescription, meaning a rule that maps the agent's private observation to an action, rather than a direct command.
In video terms, the persistent background that every segment shares is the common information, and what changes inside each short segment is private. We run one model instance per segment. Every instance reads the same cached encoding of the shared scene plus its own clip. A coordinator, which never sees raw clip content, sends all instances the same narrow, typed question with an explicit "not visible in my clip" answer. The coordinator then builds the final answer from a public record of which segment reported what. Temporal order and scene transitions are therefore read off segment indices rather than guessed by the language model. Agents never see one another's claims, so one agent's hallucination cannot spread to the others.
Core Claim
Our claim is that a large share of temporal hallucinations, specifically event-order errors, invented scene transitions, and scene-driven action guesses, come from asking one model to infer structure the input format already contains. A partition based on redundancy removes that burden. The persistent scene is computed once as common information and shared. Each segment's model answers only a closed question about its own clip. The coordinator holds only common information, namely the shared scene plus the public log of reported answers. Under this design the model never decides which event came first: it only says whether an event is visible in its clip.
We expect this to cut temporal-sequence and scene-transition hallucination at lower latency than multi-pass contrastive decoding. We also expect it to beat both free-text multi-agent debate and segment captioning followed by aggregation, with the size of the gain depending on whether the agents' questions are closed and abstention-capable.
Prior work plays three roles here. Decentralized Stochastic Control with Partial History Sharing supplies the common/private partition and the coordinator-issues-prescriptions structure. Reinforcement learning in decentralized stochastic control systems with partial history sharing supplies the observation that agents sharing identical common information compute identical prescriptions. That justifies a single coordinator with no live negotiation.
A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning (A4VL) and Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow motivate the design by documenting what goes wrong without it. A4VL's agents exchange free-text clues and vote, and can stop at a wrong answer they all agree on in the first round. In the Visual Flow study, agents trust earlier agents' text over the image, and visual attention falls by up to 60% by turn 20. Our agents exchange no text with each other and answer only typed questions, which is the contrast with both.
DyCoke supplies the method for detecting redundancy: it merges similar tokens across adjacent frames. It also reports that pruning can slightly improve accuracy. DyCoke throws the redundancy away; we keep it as shared state. That choice addresses the tension raised in VidHalluc. Its authors found that reweighting toward foreground objects fails on scene transitions, where background context is what matters.
Research Program
The system has four components.
The partition. We sample frames, cut the video into $K$ temporal segments, and merge tokens whose features stay stable across segment boundaries, using DyCoke-style cosine merging. The result is a low-resolution set of persistent tokens describing the whole video, the common scene. It is encoded once and stored as a shared key-value prefix that every agent reuses. Each segment's private view is its own clip, compressed within the clip.
The prescription family. We restrict prescriptions to a small typed set. The types are: presence of an entity or action, attribute state, motion direction, count, and "does the scene differ from the shared scene." Each type has a closed answer space that always includes "not visible in this clip." This restriction is our counterpart to the history compression used to keep prescription search tractable in Optimally Solving Dec-POMDPs as Continuous-State MDPs. We give up expressiveness so that each prescription can be checked against the clip and the search stays small.
The coordinator and public record. The coordinator is the same base model run on text only. It sees the question, a short text summary of the common scene, and the public record of (segment index, prescription, reported value). It chooses the next prescription, broadcasts it, and updates the record. Following the perfect-recall requirement of the partial-history-sharing framework, the record is append-only and built deterministically, so it never becomes inconsistent.
The answer. Relations such as "before," "after," and "the scene changes" are computed by rule from segment indices whenever the record supports it. The coordinator writes the final answer from the record and does not consult the video.
A secondary analysis addresses the open problem in SmartSight, which flags concentrated attention on a single frame as collapse. In our setting, every claim traces back to the segment that reported it. We can therefore check whether a single-model answer with concentrated attention matches what the agent for the attended segment reports. That gives a test for telling real keyframe focus from pathological fixation.
Baselines
The naive reference is the base model (Qwen2.5-VL-7B and LLaVA-OV-7B) given the full uniformly sampled frame set and asked the question directly. A matched variant applies DyCoke compression at the same total visual-token budget as our system. This separates "fewer redundant tokens helps" from "the partition helps."
The strongest existing inference-time competitor is SEASON, from SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding. It decodes against a temporally averaged negative and a noisy negative. On VidHalluc it reports +5.3% over base Qwen2.5-VL-7B and +5.6% over the trained ArrowRL baseline, up to +24.5% on the temporal-sequence subtask, and a latency increase from 0.80s to 1.21s per sample on LLaVA-OV-7B. The file reports no absolute VidHalluc score for base Qwen2.5-VL-7B, so we will re-run SEASON with its released settings on the same frames and hardware. Beating it means higher VidHalluc temporal-sequence and scene-transition scores at equal or lower per-sample latency.
We will also run DINO-HEAL from VidHalluc, which reports a 3.02% average improvement, because it is the saliency-reweighting baseline that fails on scene transitions. We will run SmartSight as the sampling-and-rejection baseline; it reports a 10.59% F1 gain on VRIPT-HAL for Qwen2.5-VL-7B.
The multi-agent competitor is A4VL. It reports 77.2% on Video-MME and 82.2% on EgoSchema, but it uses three different large backbones. For a fair comparison we will reimplement its loop of clues, cross-review, and agent pruning with our single backbone and the same number of model calls.
Two ablations isolate the key ingredients. The first replaces typed, abstention-capable prescriptions with an open "describe your clip" request; everything else stays the same. The second removes the shared common prefix, so each agent sees only its clip. All methods share frames, backbone, and prompt-tuning budget. We report wall-clock time, total tokens processed, and number of forward passes alongside accuracy.
Why not just…?
Why not just caption each segment and let a language model aggregate the captions? This is the most serious threat, and the first ablation tests it directly. Our argument against it is that open-ended captions are long generations. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that long generations drift toward the model's own text, and object co-occurrence priors of the kind documented with POPE in Evaluating Object Hallucination in Large Vision-Language Models enter that text before aggregation begins. Closed prescriptions with an abstain option keep each agent's output to a few tokens about one checkable fact. If captioning matches our method on VidHalluc and TempCompass, the gain comes from segmentation alone, not from prescriptions, and we will report that outcome.
Why not just use multi-agent debate with voting? In the Visual Flow study, agents defer to earlier agents' text instead of the image. A4VL found that plain majority voting fails when agents share biases and needed agent pruning to avoid agreeing on wrong answers. Our agents receive identical common information and never read each other's claims, so there is no channel for errors to spread. The reimplemented A4VL baseline measures what this buys at matched call count.
Why not just sample fewer frames or compress tokens harder? TempCompass shows that models working from static frames fall to 25–33% on direction and speed questions. Video-MME attributes much of the open-source gap to sampling only 8–16 frames. The DyCoke-matched baseline shows whether compression alone, at our token budget, recovers the gains.
Why not just use contrastive decoding? SEASON is effective, but it costs three forward passes per token. SmartSight notes that contrastive methods can mistake complex reasoning for hallucination. Our agents run in parallel over a cached prefix, and none of them generates long text. The head-to-head comparison on accuracy and latency settles this question.
Proposed Experiments
The primary benchmark is VidHalluc, which separates action, temporal-sequence, and scene-transition hallucination. Our prediction is the largest gains on scene transitions, the subtask where DINO-HEAL fails. TempCompass, with its reversed and concatenated videos, tests whether order derived from segment indices helps on event order without hurting within-clip direction and speed. UNSCENE tests whether the "differs from the shared scene" prescription reduces scene-driven action guesses; MASH-VLM reports 57.85% there with instruction tuning, and our method is training-free. Video-MME, split by short, medium, and long duration, and EgoSchema test whether the gains hold on long video.
Three checks decide the outcome. The first is that typed prescriptions beat open captions on VidHalluc temporal-sequence and scene-transition scores; if they do not, the central claim fails. The second is that removing the shared prefix hurts scene-transition and reference-resolution questions but not single-clip action questions, which would show the common information is doing work rather than just adding context. The third is that our system reaches at least SEASON's VidHalluc gain at no more than base-model latency. We also expect the rate of "yes" answers on absent-object probes to fall when abstention is available, and we will measure it directly.
The main risk is coverage. Questions whose answers span segment boundaries, or that the typed prescription family cannot express, will degrade. We will report accuracy separately for questions the coordinator answered from the record and questions it had to fall back on.
References
- Decentralized Stochastic Control with Partial History Sharing: A Common Information Approach
- Reinforcement Learning in Decentralized Stochastic Control Systems with Partial History Sharing
- A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
- VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding
- Optimally Solving Dec-POMDPs as Continuous-State MDPs
- SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse
- SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Evaluating Object Hallucination in Large Vision-Language Models