Boundary Assertions: Test-Time Debugging of Video Grounding Programs

Temporal grounding asks a system to find when something happens in a long, unedited video, for example "the man drops the glass" at 00:12 to 00:15. One way to do this is to have a language model write a short Python program that calls vision tools, such as detectors and captioners, over the frames. The program runs and returns an interval. These programs fail in a characteristic way. They rarely crash. Instead they run cleanly and return the wrong interval, or they return an interval for an event that never occurs. The coding-agent literature shows that models repair code well when an oracle tells them what broke, such as a compiler, a test suite, or a simulator. For grounding at test time, no such oracle exists, because the true interval is exactly what we do not know.

We propose to build that oracle out of the program itself. Each grounding program must return an interval together with executable boundary assertions. These are claims about what must be true just inside and just outside its answer, for example "a glass is in the man's hand at $t_{\text{start}}-0.5$s and not at $t_{\text{end}}+0.5$s." A separate perception model checks each assertion on a few frames. Failed assertions become a structured error report with a timestamp, for example "assertion glass in hand failed at 13.1s, inside your interval." The code model then patches the program in response. If a patch lowers the assertion pass rate, it is rolled back. If no candidate interval satisfies its own assertions, the system answers "absent." In plain terms, the model writes unit tests for its own answer, a second model runs them against the video, and the loop debugs the program until the tests pass or the model gives up.

Core Claim

Execution feedback helps temporal grounding only when it is boundary-local. It has to say where in time the answer is wrong, not merely that the code misbehaved. Our claim is that assertion-driven refinement with rollback will raise mean IoU over three alternatives at equal compute: one-shot visual programs, refinement driven by Python tracebacks alone, and best-of-N selection that uses the same assertions only as a reranker. We further expect the gains to come mainly from boundary shifts of a few seconds rather than from switching to a different event, and we expect the same mechanism to yield a calibrated "absent" decision without absence-specific training.

Relation to Prior Work

The motivation comes from two negative findings, one in each direction. On the video side, ViperGPT writes Python programs over vision APIs and runs them once. Its video programs loop over frames, but a wrong window is never revisited. LEFT grounds first-order logic programs differentiably but also fixes the program in one shot, and its temporal tests use clean motion-capture primitives rather than untrimmed video. On the code side, Yan et al. report that compiler and runtime feedback plateaus after three to five rounds because models keep making local patches. MEIC documents the "new error trap," in which an edit fixes one bug and introduces another. Taken together, these findings suggest that naively porting a traceback-driven debug loop onto ViperGPT, which is the obvious combination, would fix crashes but not intervals. We treat that port as a baseline to beat, not as the contribution.

We reuse methods from several papers. MEIC supplies the rollback rule: revert to the best stable state when a scorer regresses. CAAFE shows that raw tracebacks fed back to GPT-4 repair data-processing scripts, raising mean ROC AUC from 0.798 to 0.822. We keep that channel but add a second one. The assertion idea is closest in spirit to Two Causally Related Needles in a Video Haystack, which asks models to output precise indices as a check on retrieval. It is also close to Open-o3-Video, whose temporal gating withholds spatial reward until the timestamp is verified. Both use verification during training or evaluation. We use it as a test-time error signal that drives code edits.

Several papers constrain the design. TimeRefine already refines boundaries iteratively through predicted offsets over $K$ steps inside a Video-LLM's weights. It shows that coarse-then-refine works, so our loop must add something beyond iteration: external, inspectable checks. Agentic systems such as VideoDeepResearch and LAVE plan over fixed tools using natural-language or latent observations. Their feedback says what a tool saw, not which claim failed at which second. RaTSG shows that standard grounders invent timestamps for absent queries and handles this with a trained relevance discriminator. We test whether "no interval satisfies its own assertions" can serve as an untrained substitute.

Research Program

We will build four components. The first is a tool API: frame sampling at arbitrary timestamps, an open-vocabulary detector, a frame-level captioner and yes/no question-answering model, shot-boundary detection, and optical-flow magnitude. The second is a program schema. Every program must return an interval together with a list of assertions, and each assertion has a time offset relative to a boundary, a natural-language predicate, and an expected truth value. Assertions must sit near the boundaries, within a few seconds on each side, and include at least one "outside" check per boundary. This rule prevents trivially satisfiable claims such as "a man is present somewhere."

The third component is the verifier. It is a perception model from a different family than the program's tools, so that checks do not simply repeat the tools' errors. It evaluates each assertion on a short frame window and returns pass or fail with a confidence. The fourth is the refinement controller. For a program $P$ with assertions $a_1,\dots,a_m$, define its score as

$$S(P) = \frac{1}{m}\sum_{i=1}^{m} v(a_i),$$

where $v(a_i)\in[0,1]$ is the verifier's confidence that assertion $a_i$ holds. Each round, the code model sees the program, any Python traceback, and the failed assertions with their timestamps, and it emits a patch. A patch is kept only if $S$ does not decrease, following MEIC's rollback. The loop stops when $S$ passes a threshold $\tau$, which is tuned on a validation split, or after a fixed budget of rounds. If $S<\tau$ at the end, the system returns "absent." The schema checks that assertion count and placement stay within bounds, so the code model cannot raise $S$ by deleting hard assertions.

The system also accepts an optional initial proposal from a trained Video-LLM grounder, in which case the loop becomes a refiner on top of it. This matters because zero-shot programs will probably start below trained grounders on in-domain benchmarks. The most useful practical result may be that the loop improves a strong grounder's output, not that it replaces the grounder.

Baselines

The naive reference is one-shot ViperGPT-style program synthesis with our API. It uses no feedback and returns whatever the first program outputs. It shows what execution without refinement achieves. Every other variant uses the same code model, tools, and frame budget, so any difference comes from the feedback loop.

The strongest trained competitor is TimeRefine. It reports 47.5% mIoU on Charades-STA and 41.8% on ActivityNet, against baselines of 42.5% and 38.2% (TimeRefine). We will run it from released weights if they exist, or from a faithful reproduction whose gap to the published numbers we report. We compare in two modes. In standalone mode, our zero-shot loop is placed against TimeRefine. We do not expect to beat it in-domain, and we will say so. In hybrid mode, TimeRefine's output seeds our loop. Beating TimeRefine here means a statistically reliable mIoU gain over its own output on the same test videos, with the frame and verifier-call budget reported.

For the absence setting, the competitor is RaTSG. On Charades-RF it reports 76.85% relevance accuracy and 59.93% mIoU, against 71.94% and 56.23% for the best prior baselines (RaTSG). We compare on the same balanced present/absent pairs, with our threshold $\tau$ tuned only on the Charades-RF validation split. For multi-hop grounding, the references are GeLM-7B at 16.7 mIoU and zero-shot GPT-4o at 18.9 on MULTIHOP-EGOQA, with humans at 71.8 mIoP (Grounded Multi-Hop VideoQA). For ViperGPT-style programs on NExT-QA, no reported result found on Lacuna.

Three ablations isolate the key ingredient. The first is traceback-only refinement: same loop and budget, but the model sees only Python errors and empty-output warnings. This is the obvious combination of the two directions. The second is assertion-reranked best-of-N: sample as many independent programs as the loop would produce patches, then return the one with the highest $S$. This keeps the verifier but removes iteration. The third is refinement without rollback, which measures the new-error trap directly. We add a fourth control in which the verifier comes from the same model family as the tools, to measure how much of the gain depends on independent checking.

Why not just…?

Why not just add a traceback-driven debug loop to ViperGPT? This is the natural first idea, but the evidence suggests it fixes the wrong failures. Grounding programs usually run without error and return a plausible but wrong window, so tracebacks carry little information about the interval. Yan et al. already show plateauing after a few rounds even in pure code tasks (Yan et al.). The traceback-only ablation tests this directly. If it matches the full system on mIoU, our central claim is false.

Why not just sample many programs and pick the one the verifier likes best? This is the most serious threat. The verifier may be doing all the work, and iteration may add nothing beyond extra samples. We run best-of-N at matched compute, counting total code-model tokens and verifier calls. We also break results down by how far the first program's interval is from ground truth. Iteration should matter most when the first program found the right event with loose boundaries, because local assertion failures then point to the needed shift. Resampling would have to rediscover that event from scratch. If best-of-N matches the loop across these strata, we will report the contribution as an assertion-based reranker and drop the claim about refinement.

Why not just fine-tune a Video-LLM with offset refinement, as TimeRefine does? On in-domain benchmarks, a trained grounder will probably beat a zero-shot program loop, and we do not argue otherwise. The case for our loop rests on three things weight-bound refinement cannot provide: inspectable reasons for each boundary move, rejection without absence-specific training, and composition across scattered windows. The hybrid mode tests whether the two are complementary. If assertion refinement cannot improve TimeRefine's outputs, and if it does not help on Charades-RF or MULTIHOP-EGOQA, then the fine-tuned model alone is the better answer.

Why not just ask a vision-language model to critique the interval in words? Agent systems like VideoExplorer already loop on natural-language observations. We add a critique variant in which the verifier writes a free-form judgment of the interval instead of checking assertions. Our hypothesis is that critiques without timestamps produce vague edits that change which event is chosen rather than tightening boundaries. The fraction of rounds that improve IoU, compared between the two variants, measures this.

Proposed Experiments

The main experiments cover Charades-STA and ActivityNet Captions in both standalone and hybrid modes, measuring mIoU and recall at IoU 0.5 and 0.7. Charades-RF covers presence and absence, measured by relevance accuracy and mIoU. MULTIHOP-EGOQA is a stretch setting where a program returns several intervals, each with its own assertions.

Beyond the headline metrics, we log three quantities per refinement round. The first is the fraction of accepted patches that raise IoU. The second is the fraction that lower it, which is the regression rate with and without rollback. The third is the correlation between the change in $S$ and the change in IoU. This correlation is the core validity check. If raising assertion pass rates does not track IoU on held-out data, the oracle is not measuring what we need, and every downstream gain is suspect.

Three results would settle the question. First, the full loop must beat both traceback-only refinement and compute-matched best-of-N on Charades-STA mIoU. Second, in hybrid mode it must improve on TimeRefine's own output. Third, the "absent" decision derived from $S$ should approach RaTSG's 76.85% relevance accuracy on Charades-RF without being trained for relevance. If the first holds but the second does not, the method is a zero-shot tool rather than a refiner. If only the reranking ablation holds up, the assertions are the contribution and the debug loop is not.