Physical Alibis as Verifiers for Small Video Reasoning Models

Video question-answering models increasingly "show their work." They emit claims such as "the red cup, at this box, at 3.2 seconds" inside their reasoning, then answer from those claims. When a scene holds several look-alike objects, small models often stitch together a chain that no physical object could have produced. The cup "jumps" eight body-widths in a third of a second, or an object reappears on the far side of the frame after an occlusion that allowed no time to get there. Multi-camera tracking solved a version of this problem long ago with "alibi" checks: two sightings cannot be the same individual if the distance between them could not be covered in the time between them. This proposal turns that old check into a calibration-free verifier. It reads a video model's grounded reasoning chain, flags identity links that are physically impossible, and returns a concrete counterexample along with the feasible alternatives so the model can repair the chain.

The broader question is one that agent research keeps hitting. Wrapping a weak base model in debate, reflection, or delegation mostly reorganizes what the model can already do, and the gains shrink as the model gets weaker. We hypothesize that an external check is different when it supplies a fact the model cannot compute itself, such as whether a displacement is physically possible, rather than another round of the model's own judgment. If so, its gains should be largest for the smallest models, which is the opposite of the scaling pattern reported for scaffolds. As a by-product, the verifier splits a model's failures into two kinds: chains that bind the wrong object, which the verifier catches, and chains that are physically consistent but still lead to a wrong conclusion, which it does not. That split is the cleanest measure we know of where perception ends and the reasoning bottleneck begins.

Core Claim

A cheap, training-free feasibility filter in image space can serve as a symbolic verifier for grounded video reasoning. The filter checks apparent displacement against apparent object size, compensates for camera motion, and suspends itself across shot cuts. When its violations are fed back as structured counterexamples, it should improve answer accuracy and grounding on questions that require tracking identity among look-alike objects. The improvement should be larger at 3B than at 32B or more. On physically consistent but wrong chains, the filter should give no improvement, and that residual estimates the true reasoning ceiling.

What Each Direction Contributes, and What Is Already Done

Spatio-temporal constraint filtering supplies the mechanism. Santore and Shapiro frame the core problem as perceptually indistinguishable objects. An appearance-only agent cannot tell a re-sighting from a twin unless it reasons about whether the displacement is feasible. They implement that reasoning with speed limits in a logic system, but it depends on the observer's own localization. Tracking systems make the idea practical. Ren et al. project boxes to world coordinates and prune matches outside travel-time windows. Kondor et al. name this the alibi constraint. Hsu et al. and TFusion (Lv et al.) learn transition statistics from data rather than hand-setting them. All of these rely on calibrated cameras or fixed infrastructure, which is the main open limitation of the area. They also treat constraints as post-processing for a tracker, never as feedback to a reasoner. We reuse the learned-statistics idea but move it into normalized image coordinates, so it applies to arbitrary internet video.

The base-model bottleneck literature supplies the motivation and one constraint on design. ReDel finds that the margin of agentic systems over single agents grows with base-model capability. He et al. find that debate collapses into shared misconceptions when backbones are weak. Both results argue against adding more model-judged scaffolding. SymDiag is the closest methodological precedent. It translates text reasoning into Prolog, separates translation errors from reasoning errors, and feeds formal counterexamples back for repair. It reports 70.7 faithfulness F1, against 62.8 for LLM-as-judge and 57.7 for answer matching. But it works only on text math and logic, with no space, time, or video. We transplant its counterexample-driven repair loop into grounded video, where the "proof check" is a physical feasibility test rather than a logic engine.

The closest video work stops short of this. Chain-of-Glimpse runs tree search over object boxes across frames with a GRPO-trained 3B/7B policy. Its branches are scored only by learned priors, and its authors note the ceiling set by the small backbone. Open-o3-Video emits box-and-timestamp evidence in a single pass and uses confidence-aware self-evaluation at test time. It cannot reject kinematically impossible evidence, and it degrades on cluttered scenes and small objects. MUPA verifies by consensus over temporal intervals only. NeuS-QA uses formal temporal-logic model checking but works at whole-frame level, with no boxes or displacement. STL-GO encodes spatio-temporal and topological constraints for solvers but never touches pixels or language. None of these checks the physical consistency of a model's own grounded identity claims and returns the violation to the model. That is the gap we target.

Research Program

The verifier works on the evidence chain a grounded model already produces. This is a sequence of object mentions with boxes and timestamps, linked by the model's assertion that they refer to the same entity. For two linked sightings of an object of category $k$, at times $t$ and $t'$ with box centers $c_t, c_{t'}$ and box areas $s_t, s_{t'}$, we compute a size-normalized displacement after removing estimated camera motion:

$$d = \frac{\lVert H_{t \to t'}(c_t) - c_{t'} \rVert}{\sqrt{\sqrt{s_t s_{t'}}}}, \qquad \text{feasible iff } d \le \kappa_k \,|t'-t| + \epsilon .$$

Here $H_{t \to t'}$ is a frame-to-frame homography estimated from background feature matches. It is a camera-motion estimate, not a calibration. $\kappa_k$ is a per-category speed bound in "object widths per second," and $\epsilon$ absorbs box jitter. Normalizing by apparent size is the key move. It stands in for depth, so a car far away and a car nearby share one bound, and no world coordinates are needed.

We estimate $\kappa_k$ as a high quantile of the same quantity, measured over ground-truth tracks in public tracking data. This is the image-space analogue of the learned transition statistics in Hsu et al.. The check is suspended across detected shot cuts. A second, weaker constraint covers occlusion gaps: an object that leaves the frame on one edge cannot re-enter on another edge sooner than the bound allows. This borrows the tripwire idea of Ren et al. without fixed lines.

When a link fails, the verifier writes a structured counterexample. It names the two claimed sightings and the implied speed against the bound. It also lists the detector boxes at $t'$ that would be feasible continuations, keeping all of them rather than committing to one, in the spirit of Santore and Shapiro's set-valued identity hypotheses. The model gets at most two repair rounds. Chains that remain infeasible are down-weighted when votes over sampled chains are aggregated.

We apply this unchanged to Qwen2.5-VL at 3B, 7B, and the largest size our compute permits, and to Open-o3-Video and Chain-of-Glimpse checkpoints where available. Inside Chain-of-Glimpse's tree search, the same test acts as a hard pruning rule on expansions. That lets us compare pruning during search with repair after generation.

The diagnostic analysis labels every wrong final answer by whether its chain contained a feasibility violation. On STSBench, ground-truth 3D trajectories exist, so we can also check the filter's precision against true identity. This addresses the tension between STSBench, which finds perception to be the bottleneck, and Video-Holmes, which attributes about 60% of failures to reasoning and about 35% to visual omission. We expect the split to depend on whether the question requires tracking identity, and we will measure it per question type.

Baselines

The naive reference point is the base model answering directly: Qwen2.5-VL at each scale, with the same frame budget. On NExT-QA, Chain-of-Glimpse reports 79.7% accuracy for the 7B base. That number anchors our reproduction.

The strongest grounded competitor is Open-o3-Video with its own confidence-aware test-time scaling, run with the same number of sampled chains as our method. Same samples, same frames, our filter added or removed: this isolates the verifier. Open-o3-Video reports only gains on V-STAR over Qwen2.5-VL, +14.4 mAM and +24.2 mLGM. No absolute V-STAR value for it appears in our file, so we will reproduce it ourselves. Beating it means a significant gain in mLGM, its grounding-sensitive metric, and in accuracy on our look-alike subset, at equal sample count.

The strongest search competitor is Chain-of-Glimpse, which reports 83.3% on NExT-QA and 42.5% on Video-Holmes at 7B. We run it with and without feasibility pruning, with node budgets matched to equal wall-clock time.

On grounded QA, MUPA-7B's reported 30.3% Acc@GQA on NExT-GQA is the bar to beat (MUPA). We also run a tracker-to-text pipeline: an off-the-shelf multi-object tracker whose tracks are serialized as numbers and given to the same language model, mirroring the STSBench finding. There, text-only GPT-4o on ground-truth coordinates reaches 57.08%, while InternVL 2.5 8B on pixels reaches 46.07% (STSBench).

Three ablations isolate the key ingredient. The first replaces the verifier with an LLM judge that gives free-text critique over the same number of repair rounds. The second rejects links at random, matching the verifier's rejection rate, to rule out benefits from resampling alone. The third keeps the counterexample format but sets $\kappa_k$ to infinity, so the model receives structure without physics. Beating the baselines means the full verifier beats all three ablations on the look-alike subset at matched compute.

Why not just…?

Why not just use a bigger base model? Scale helps but does not close the gap on identity-heavy reasoning. Frontier models score 20% to 32% on EgoSchema against 76.2% for humans, and Gemini-2.5-Pro reaches only 45.0% on Video-Holmes. Our scale sweep tests this directly. If the filter's gain at 3B matches or exceeds the jump from 3B to the largest model on the look-alike subset, the verifier is a cheaper substitute for scale. If its gain vanishes at large scale, larger models already perform this check internally. Either result is informative.

Why not just sample more chains and vote, or have models debate? Voting amplifies a shared binding error when most samples latch onto the same salient look-alike. DebateCV documents conformity and a "neutrality trap" in moderators. In SymDiag, answer matching and LLM judging both trail formal verification on faithfulness. Our matched-sample self-consistency and LLM-judge ablations measure this in the video setting.

Why not just run a tracker and hand its tracks to the language model? This is the real threat. Trackers with motion gating already enforce continuity, and STSBench shows that text reasoning over coordinates can beat pixel reasoning. Where we expect it to lose is the regime the filter targets: occlusions and look-alikes, where a tracker commits to one association and silently swaps identities. The verifier instead keeps every feasible continuation and lets the model's semantic understanding choose among them. We will report results separately on occlusion and look-alike questions. If tracker-to-text matches us there, the honest conclusion is that perception pipelines, not verification, are the fix, and we will say so. We also test the hybrid that gives the model both serialized tracks and verifier feedback.

Why not just train the physics in as a reward? A kinematic term could be added to Open-o3-Video-style reinforcement learning. That is plausible, and we will run it as a variant at 7B. The test-time verifier needs no retraining and applies to any grounded model. It also yields the error diagnosis, which a trained-in reward hides. If the reward variant wins, the filter still has value as the reward function.

Proposed Experiments

The evaluation covers V-STAR for joint spatial and temporal grounding, NExT-GQA and NExT-QA for grounded and causal QA, and Video-Holmes for complex reasoning. It also includes STSBench, where ground-truth trajectories let us score the filter's own precision and recall against true identity. As a sanity check that the image-space bound works at all, we run it as a pruning step on CityFlow single-camera association. There, visual ReID alone reaches 32.0% mAP and a calibrated spatio-temporal model reaches 46.3% ID F1. Recovering a meaningful part of that gap without calibration would validate the normalization.

Because no existing benchmark isolates perceptually indistinguishable objects, we will build a look-alike subset. It draws questions from V-STAR and NExT-GQA whose answers depend on an object with at least one same-category distractor present. It is supplemented by a small set of procedurally generated "shell game" clips with known ground truth.

Three checks decide the outcome. First, the precondition: at least roughly one in ten wrong chains on the look-alike subset must contain a detectable violation, verified against ground truth where available. If not, the binding-error hypothesis is wrong and we stop. Second, the verifier must beat the three ablations at matched compute, showing that physics, not structure or resampling, drives the gain. Third, the gain must be larger at 3B than at the largest scale, with no gain on physically consistent wrong chains. That pattern would show that a verifier supplying missing facts escapes the scaffold scaling pattern of ReDel, while measuring how much of the remaining error is the reasoning bottleneck itself.