Right Answer, Wrong Evidence: Step-Checked Training Traces for Video Program Repair

A growing family of video question-answering systems no longer answers directly. Instead, the model writes a small program that calls vision tools (find the frame where the ball hits the wall, count the red objects after second 12), runs it, reads any errors, and rewrites the program until it produces an answer. These systems are trained on synthetic "reason-and-refine" transcripts produced by a stronger model, and those transcripts are kept or discarded based on one check: did the final answer come out right? This proposal argues that this check is the wrong filter. In a repair loop, a program can reach the correct answer while its intermediate tool calls are hallucinated, or after a "fix" that quietly stops using the video at all. One example is an exception handler that returns a default guess. Training on such transcripts teaches models to make repair loops converge on plausible answers, not on evidence. Because many video benchmarks can be partly solved from text priors or leaked test items, this failure is hard to see on the benchmarks people currently report.

We propose to generate videos from scripts, so the full scene state (object identities, positions, event timestamps) is known exactly for every frame. That known state lets us check every intermediate step of a synthetic repair transcript, not just its last line. Training traces are then filtered by whether each tool output and each repair matches the rendered truth. Evaluation is designed to separate real temporal evidence use from memorization: freshly regenerated scene variants with predictably changed answers, frame-scrambling controls, and transfer to natural-video benchmarks. The core bet is that "evidence-correct" traces teach repair loops that are more honest and more robust than "answer-correct" traces, at equal data and compute.

Core Claim

Among synthetic visual-program repair transcripts that end in a correct answer, a substantial fraction reach it through steps that disagree with the video. We will report what that fraction is rather than assume a value. Filtering at the step level, rather than at the answer level, should produce models whose answers actually depend on temporal evidence. That dependence can be measured: the answer should change when the evidence frames are scrambled or removed. These models should also hold up better on unseen, regenerated variants of the task.

The new piece is not programmatic video, which MORSE-500 already uses to build a fixed set of 500 scripted test clips. Nor is it video program repair, which VideoPro already trains on GPT-5-synthesized transcripts. The new piece is using the renderer's ground truth as a step-level oracle for synthetic repair data. Its quality is then measured by whether answers depend on evidence and whether they survive regeneration, not by static benchmark accuracy.

Where This Sits

The motivating evidence comes from three places. VideoPro routes low-confidence queries to multi-step visual programs, repairs them from execution logs, and is trained via supervised fine-tuning followed by reinforcement learning on synthetic transcripts. It treats those transcripts as clean, and it notes that noisy tool outputs can make the loop fail silently. Code-synthesis work shows that unverified generated code and tests carry structured noise: CodeEvo filters with compilers and tests in a refine-and-retry loop, and ACECODER documents hallucinated test assertions that corrupt reward signals. Both verify outcomes against tests; neither has a way to verify the perceptual intermediate steps a video program depends on. Finally, iterative repair itself degrades code. SlopCodeBench finds structural erosion rising in 77% of agent trajectories and verbosity in 75.5%. Yan et al. report that feedback gains plateau after 2–4 iterations, and Srivastava et al. find that repeated prompting can entrench errors. These results also constrain the design: we cannot assume more refinement is better, so refinement depth is a variable we measure.

The method borrows from three sources. From Temporal-Aware Reasoning Optimization we take the idea that scrambling frames near event boundaries exposes temporally agnostic models. TaRO uses this as a training reward for timestamp prediction; we use it as an evaluation probe on program outputs. From CausalVQA we take a blind text-only filter, which removed 39% of its candidate questions, and paired scoring. From DyCodeEval we take metamorphic variants: tasks with the same underlying logic but altered surface content, which exposed fine-tuning on leaked data (Llama-3.2-1B rose from about 20% to about 80% on the static set but stayed near 13% on the variants). DyCodeEval rewrites text with LLM agents and warns of "information overload." We instead change scene parameters inside the renderer, so a variant cannot drift from its ground truth. Logic-in-Frames verifies logical constraints with an object detector during frame search. It is the closest prior work on execution-checked temporal evidence, but it verifies at inference time against a detector, not against ground truth, and it does not touch training data.

Contamination enters for the reason Both Text and Images Leaked! gives: text leakage lets multimodal models answer without looking at the visual input. LiveCodeBench shows the size of the resulting gap for code: DeepSeek-Coder-33B trails GPT-4 by 4.3 points on HumanEval+ but by 16.2 on post-cutoff problems. A video repair loop whose fallback is a language-prior guess will look strong exactly where leakage is high. That is why our evaluation has to be regenerated rather than static.

Research Program

Scene families. We will build scene families in the style of MORSE-500 using Manim and MoviePy: collisions, ordering of appearances, counting under occlusion, and cause-before-effect chains. Each family is a script whose parameters control object counts, timings, colors, and distractors. Every render logs a per-frame scene state. Each family has a question template whose answer is computed from the state, so changing a parameter changes the answer in a known way.

Synthetic transcripts. We will generate repair transcripts with a strong teacher model using a VideoPro-style tool library: temporal localization, frame retrieval, detection, and counting. The tools run on the rendered pixels, so they are realistically noisy.

Step checker. A checker compares each tool call's output to the logged scene state. It also flags three kinds of repair edits that bypass evidence:

  • hard-coded answers;
  • exception handlers that return defaults;
  • branches that never read a tool output.

These are the "fallback slop" analogue of SlopCodeBench's verbosity metric. The checker labels each transcript as evidence-correct, answer-correct-only, or wrong.

Training regimes. We train a Qwen3-VL-8B student, the same base VideoPro reports, under three filters with the trace count held equal: answer-correct traces, evidence-correct traces, and unfiltered traces. A fourth arm trains on evidence-correct traces with injected tool noise, to test whether the student learns to repair around a faulty tool rather than trust it.

Analysis. We will measure the answer-correct-only fraction as a function of refinement depth. Our hypothesis is that it grows with each repair round, which would offer one mechanism for the 2–4 iteration plateau.

Baselines

Naive reference. The student model answering directly from sampled frames, with no programs and no fine-tuning. It sets the floor and, through the blind and scramble probes, shows how much of its accuracy needs no temporal evidence.

Strongest existing method: VideoPro's recipe. This is supervised fine-tuning on answer-filtered reason-and-refine transcripts, reproduced on our rendered training families with the same base model, tool library, number of traces, training steps, and per-query refinement budget as our arm. VideoPro reports raising Qwen3-VL from 61.2% to 65.0% after supervised fine-tuning alone, and an average gain of 6.7% for Qwen3-VL-8B on long-video benchmarks. We found no reported result on Lacuna for VideoPro on MORSE-500 or CausalVQA, so we will run it ourselves. On natural video it would be beaten if our arm matches its LongVideoBench and LVBench accuracy within noise while showing significantly higher evidence dependence and regenerated-variant accuracy. A drop of more than a couple of points on natural video would count as a loss, whatever the probes show.

Strongest non-program competitor: TaRO-style training. A temporal-sensitivity reward trained on the same base model and data budget. This is the strongest approach that skips programs entirely, and it reports +8.4% [email protected] on TVGBench over Time-R1.

Frame-search reference. Logic-in-Frames plugged into the same answering model. It reports an 8.7% gain for GPT-4o on LongVideoBench.

Reference points on rendered and egocentric video. On rendered video, the cited ceiling is OpenAI o3 at 23.6% versus 55.4% for humans on MORSE-500. On egocentric video, Gemini 2.5 Flash reaches 61.66% versus 84.78% for humans on CausalVQA.

Key ablation: answer-filtered versus evidence-filtered. This compares the answer-filtered and evidence-filtered arms at identical size. Any difference between them can only come from what the step checker adds.

Why not just…?

Why not just filter by final-answer correctness, as CodeEvo does with tests? This is the main threat, and the key ablation tests it directly. For code, a passing test is strong evidence that the program is right. For video, the "test" is a single answer, often multiple choice, which a language prior or a default fallback can hit by chance. If answer-filtered and evidence-filtered arms are indistinguishable on scramble sensitivity and regenerated-variant accuracy, the claim fails and we will report that.

Why not just skip programs and train the video model directly with a temporal reward like TaRO? That may well suffice for timestamp grounding, where TaRO's gains are large. But a direct model gives no inspectable intermediate steps, and TaRO itself documents rationales that contradict final predictions. We include TaRO-style training as a baseline. If it equals our arm on evidence dependence and natural-video transfer, the program route offers no added value on these tasks.

Why not just evaluate on a fixed scripted benchmark like MORSE-500, or apply CausalVQA's blind filter? A fixed public set will eventually be scraped into training data, and a blind filter removes text-solvable items but cannot detect a model that memorized the specific video-answer pairs. Regenerated variants with predictably shifted answers catch both. We will report static and regenerated accuracy side by side. A gap that grows over training would show the static set rewarding memorization.

Why not just cap refinement at two or three rounds? Capping limits the damage but does not remove bad traces from the training data. We will train capped answer-filtered models as an extra arm. If the cap alone closes the gap to step-checked training, the cheaper fix wins.

Proposed Experiments and Decisive Checks

Training and test families. Train on rendered families and test on held-out families that were never released, plus parameter-regenerated variants of the training families. Score with paired, all-variants-correct accuracy in the spirit of DyPass@K.

Evidence dependence. Measure the fraction of correct answers that flip under two interventions: scrambling frames near the ground-truth event, and deleting the evidence frames. Report alongside this the blind-text accuracy, and the rate of evidence-bypassing repair edits in the model's own refinement loops, scored with the step checker.

Transfer. Test on CausalVQA, LongVideoBench, and LVBench to check whether training on rendered video generalizes or merely learns the regularities MORSE-500 warns about.

Decisive outcomes. The proposal succeeds if three things hold at matched data and compute:

  • evidence-filtered training beats answer-filtered training on regenerated-variant accuracy and on scramble sensitivity;
  • it is no worse on natural video;
  • the answer-correct-only fraction of teacher transcripts measurably rises with refinement depth.

It fails if the answer-correct-only fraction is small, or if the two filters produce indistinguishable students. Either outcome would mean answer-level filtering is adequate for video program repair, which is itself useful to know, since current pipelines rely on it.