Postdiction-Guided Re-Looking: Correcting Hallucinated Steps in Video State Tracking

When a vision-language model watches a video of cups being shuffled or tiles being slid and has to say where things started, it usually gets the answer wrong. The reason is less often faulty logic than a hallucinated step early on: a swap that never happened, a tile moved the wrong way. Once that error is in the model's running account, nothing later corrects it. Classical AI has a name for the missing ingredient. Retrospective belief revision, also called postdiction or smoothing, uses what is observed later to go back and fix what was believed about earlier moments. We propose a simple test-time procedure built on it. The model first writes out its step-by-step account of the video as an explicit list of operations and states. A backward pass then replays that list from the end and finds the steps that cannot all be true together. The model is sent back to re-watch only those few moments, with pointed yes/no questions about what should be visible there.

The bet is that the backward pass is a cheap and precise way to locate errors. It is not a reasoning engine. Re-perceiving a handful of well-chosen frames should beat re-perceiving everything, sampling many answers, or thinking longer, at matched compute. The benchmark evidence says the bottleneck is perception, not deduction. On VideoReasonBench, replacing the video with a ground-truth text description raises Seed1.5-VL from 11.5% to 69.4%. Models are markedly worse when the state is revealed at the end and must be inferred backward. A method that only reasons harder over a corrupted account cannot close that gap. A method that uses reasoning to decide where to look again might.

Core Claim

Treat the model's forward account of a video as a belief trajectory: a sequence of perceived operations $a_1,\dots,a_T$, each with a confidence, plus implied intermediate states. Revise it the way formal belief revision prescribes. Choose the smallest, least-confident set of step changes that makes the trajectory consistent with later evidence, then verify those changes against pixels before accepting them. Concretely, the backward pass searches for a revised sequence $\hat a_{1:T}$ minimizing

$$ \sum_{t=1}^{T} \kappa_t \,\mathbb{1}[\hat a_t \neq a_t] \quad \text{subject to } \hat a_{1:T} \text{ being legal and consistent with all observed states}, $$

where $\kappa_t$ is a disbelief cost for changing step $t$, set from the model's negative log-probability of its original reading. "Legal" means each move is physically possible given the state it acts on; a slide must move a tile adjacent to the blank. "Observed states" are any frames where part of the state is directly visible, including the final reveal. The cost $\kappa_t$ plays the role of Spohn-style plausibility ranks in Boutilier's Generalized Update, which formalizes how a late observation forces abductive revision of earlier assumptions. Minimal change is the principle behind the distance-based belief-evolution operator of Hunter et al., which traces contradictions back through an action sequence to the initial state.

The steps the solution changes become concrete visual claims ("between 0:12 and 0:15, did the left and middle cups swap?"). The model re-answers each claim while viewing only that clip, at higher frame density. A revision is accepted only if re-perception confirms it. This last part is what separates the method from both a pure symbolic solver and a pure self-critique loop.

Relation to Prior Work

Three hallucination-mitigation methods already look backward, and each motivates the gap. OPERA rewinds beam search a few tokens when attention collapses onto summary tokens. Its rollback is triggered by an attention statistic and never knows which fact became contradictory. VBackChecker grounds each claim of a finished description back into the image with a segmentation decoder. It is post-hoc, static-image only, and does not feed failures back into generation. Volcano critiques and rewrites whole responses in natural language. It roughly doubles latency (about 5.8s versus 2.7s per response) and needs a separate decide stage to catch errors its own revisions introduce. Our proposal differs from all three in one way: the unit of revision is a structured state trajectory, and errors are localized by a consistency check over that trajectory rather than by attention heuristics or open-ended critique. We reuse VBackChecker's idea of turning claims into targeted verification queries, now applied to temporal operations instead of objects.

The method's ingredients come from symbolic smoothing. The Backwards-Forwards algorithm of Mombourquette et al. regresses recent observations to infer what must have held earlier. It reaches 100% coverage where forward-only approximate filtering gets 0% on Wumpus05, and tracking only relevant variables cuts the state to under 1% of fluents. That result is our reason to expect a backward pass to recover information that forward tracking discards. The differentiable two-filter smoother of Learning to be Smooth and the retrospective trellis tracker of Self-Paced Learning for Long-Term Tracking show the same pattern in perception: revisiting earlier ambiguous frames once later evidence arrives cut tracking error by a factor of four. Both work in low-dimensional continuous state. We use discrete operation sequences, where exact minimal-change search is cheap, and so avoid the dimensionality problems those papers report.

Why LVLMs Are More Prone to Hallucinations in Longer Responses supplies the failure mechanism we expect. As generation lengthens, attention drifts from visual tokens to prior text, so a long forward narration of a video should accumulate exactly the step-level errors we target. Two papers constrain the design. VideoReasonBench shows that dropping half the frames costs 55% of performance, so the re-look step must add frame density, not merely re-read the same frames. Yan et al. show that models often reach right answers through fabricated premises, so we score step-level correctness, not only final accuracy.

Research Program

The work runs in three stages. The first stage instruments the forward pass. An open model (Qwen2.5-VL-72B, plus a 7B variant for cheap ablations) is prompted to emit a fixed-schema trajectory of operation per segment, implied state, and token-level confidence for each operation. We validate on VideoReasonBench's synthetic categories (number puzzles, circles, cups, files, cards and chips), which have deterministic, enumerable operations. Where operation-level ground truth is not released, we annotate operations for a 300-question subset; this costs little because the videos are procedurally generated.

The second stage builds the backward solver. For each category we write a small transition model: legal moves, plus which state components are visible in which frames. We then solve the weighted minimal-edit problem above by bounded search. Edits are capped at three steps, since sequences run 5 to 14 operations, which keeps the search exhaustive. The output is a ranked list of suspect steps with proposed alternatives.

The third stage is re-perception. Each suspect step becomes a binary question posed on the corresponding clip at four times the original frame rate. Answering in yes/no form inherits POPE's finding of stable binary probing. It also inherits POPE's "yes" bias, so we ask both the original and the proposed alternative and accept only asymmetric answers. Confirmed edits update the trajectory, the answer is recomputed by deterministic replay, and the loop repeats at most twice.

Baselines

The naive reference point is the base model answering directly, and then with a free-form chain of thought, at the frame rate VideoReasonBench uses. Published reference numbers on the full benchmark are 6.9% for GPT-4o, 7.2% for Qwen2.5-VL-72B, 56.0% for Gemini-2.5-Pro, and 73.8% for humans, all from VideoReasonBench. Our primary target is the open model we can instrument, where 7.2% is the number to beat. Gemini-2.5-Pro is reported as a ceiling reference, not a fair competitor.

The strongest existing competitor for this question is test-time scaling. VideoReasonBench reports that a larger thinking budget raised Gemini-2.5-Pro by 9 points. We run Qwen2.5-VL-72B with long-form reasoning given the same total generated-token and visual-token budget our method consumes, including re-looks. Beating it means a higher accuracy on backward-reveal questions at equal or lower compute.

The second strong competitor is compute-matched self-consistency. We sample $K$ forward trajectories at the same total frame budget, with $K$ chosen to match our cost, and take a majority vote over final answers. A second variant first filters out trajectories that violate the legality constraints. This is the baseline most likely to match us, because sampling also corrects random step errors.

The third competitor is a Volcano-style critique-and-revise loop. It cannot be run off the shelf: Volcano's released models target static images and no reported result found on Lacuna for Volcano, OPERA, VCD, SHARP or RITUAL on VideoReasonBench. We therefore implement the critique-revise-decide prompting protocol on the same base model and budget. OPERA's rollback can be applied directly to the narration decoding and serves as the token-level retrospection reference.

Three ablations isolate the key ingredient. "Solver-only" applies the minimal-edit revision without re-perception, accepting edits on consistency alone. "Uniform re-look" re-perceives the same number of steps chosen at random. "Confidence-only re-look" re-perceives the lowest-confidence steps without the consistency solver. An oracle that re-looks exactly the truly wrong steps gives the upper bound. The proposal counts as successful only if the full method beats every ablation and both compute-matched baselines on backward-reveal accuracy, with nonoverlapping bootstrap intervals over questions.

Why not just…?

Why not just let the model think longer? VideoReasonBench already shows that extra thinking helps only when perception is sound, and the 11.5%-to-69.4% text-substitution result says the missing information is visual. Longer reasoning over a wrong operation list stays wrong. Still, Gemini's 9-point gain shows that reasoning is not useless. The compute-matched long-reasoning baseline settles this directly. If it matches us on backward-reveal questions, our localization adds nothing.

Why not just sample many answers and vote? This is the real threat. If step errors are independent across samples, voting fixes them without any solver. Our reason to doubt it is that the errors come from systematic perceptual and prior-driven confusions (the long-response attention drift in Why LVLMs Are More Prone...), so samples share them, and resampling at the same frame rate never gains visual detail. The constraint-filtered self-consistency baseline removes the legality advantage from the comparison. If it still matches us, we will report that the value lies in the constraints, not the targeted re-look, and change the claim accordingly.

Why not just re-watch the whole video at higher frame rate? That spends compute uniformly where errors are concentrated in a few steps. The uniform re-look and full-dense re-perception baselines measure exactly this trade. We expect dense re-perception to be competitive at much higher cost. The decisive comparison is accuracy per visual token.

Why not just use the symbolic solver and skip re-perception? When several minimal edits are equally consistent, the solver cannot choose among them, and it will confidently "fix" a correct step to match a hallucinated one elsewhere. The solver-only ablation measures how often this happens. We expect it to help most in the sliding-puzzle category, where legality is highly constraining, and least for shell games, where many swaps are legal.

Proposed Experiments

The main experiment reports accuracy on VideoReasonBench by category and by reasoning level (recall, infer, predict), split into state-revealed-at-start and state-revealed-at-end questions. The prediction is a much larger gain on end-reveal questions, which is where postdiction has information to exploit. A second measurement is step-level operation hallucination rate on the annotated subset, before and after revision. It tests whether improvements come from fixing perceived steps rather than lucky final answers. Localization quality is scored as precision at $k$ of flagged steps against true erroneous steps, comparing solver-ranked, confidence-ranked and random selection. If the solver's precision is not clearly above confidence-only, the central claim fails regardless of accuracy. Compute is reported as visual tokens, generated tokens and wall-clock per question for every method, following the latency reporting of SHARP and Volcano.

Two checks would decide the matter. First, the method must close a meaningful fraction of the gap to the oracle re-look, while compute-matched self-consistency does not. Second, gains must shrink when the final-state reveal is removed from the video, since that is the later evidence the backward pass relies on. A method that improves equally without that evidence is succeeding for some reason other than retrospective revision, and we would say so.