Which Caption Errors Are Inherited? Counterfactual Prefix Repair for Video Narratives

Many video-language models now describe a long video one event at a time, and each new description is written while the model reads its own earlier sentences. If the model misreads the first event, for instance calling a whisk a spoon, later sentences can repeat and build on that mistake even when the video plainly contradicts it. Current metrics hide this. Word-overlap scores such as CIDEr, and even recent judges built on large language models (LLM judges, which grade text by reasoning about it rather than counting shared words), give one flat score or an additive list of errors. A model that makes one perception error that then spreads looks the same as a model that makes five unrelated perception errors. These two models need different fixes: better visual grounding in the first case, better resistance to its own history in the second.

We propose to separate the two causally rather than by inference. A multimodal judge, one that watches the video frames, finds the erroneous event descriptions and writes a minimal correction for each one. We then re-run the captioner from that point with the corrected history in place of its own. Errors that disappear after the repair are counted as inherited from the repaired error. Errors that survive are counted as local. The result is a per-error propagation score and a per-model "inheritance rate," measured by intervening on the model instead of asking a judge to guess dependencies from text. A second question comes out of the same setup almost for free: do judges themselves snowball, becoming lenient toward later hallucinations that agree with an earlier one they have already read?

Core Claim

Video narratives produced autoregressively, with each event description conditioned on the previous ones, contain a substantial and model-dependent share of errors that are inherited rather than perceived. Re-running the captioner on judge-repaired prefixes measures that share more accurately than any text-only attribution. For an error claim $e$ at event $j$ and an upstream erroneous event $k<j$, we define its inheritance as

$$I(e \mid k) = P(e \text{ at } j \mid \text{original prefix}) - P(e \text{ at } j \mid \text{prefix with event } k \text{ repaired}),$$

where each probability is estimated over several sampled re-generations with the same video input. The probability of $e$ under the original prefix is not taken from the single logged output. It is re-estimated by re-running the original prefix the same way, which works as a placebo control for sampling noise. A model's inheritance rate is the fraction of its errors with $I$ significantly above zero.

Relation to Prior Work

The motivation comes from three findings. In agentic text workflows, AgentEval builds a graph of execution steps, judges each step with GPT-4o, and reports that 63% of step-level failures propagate from upstream rather than arising locally. It covers no visual inputs, and it attributes root causes through a judge reading the trace, which reaches 72% root-cause accuracy against an 81% human ceiling. We keep its step-level view but replace judged attribution with intervention, which video captioning makes cheap: unlike a deployed agent, the captioner can be re-run on the same clip. The second line of motivating work comes from Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models and MM-Snowball. These show that vision-language models override visual evidence to stay consistent with corrupted dialogue history; for example, LLaVA-1.5-7B falls below 2% accuracy in turns 3 to 5. Both studies use static images and short question turns, not continuous video narration. Third, Streaming Dense Video Captioning feeds earlier generated captions back as a prefix, which raises ActivityNet CIDEr from 30.2 to 41.2. It is trained on ground-truth prefixes and scored only by n-gram overlap, so whether that prefix also carries errors forward has never been checked.

Other work supplies parts of the method. ARGUS uses GPT-4o for sentence-level entailment on dense captions and aligns sentences with dynamic programming that penalizes hallucinations and temporal inversions. We reuse its sentence-level error detection, but ARGUS adds penalties independently and never asks whether one error caused another. VideoJudge shows that a 3B judge that writes an instance-specific rubric before scoring cuts mean absolute error from 1.15 to 0.59. It also shows that text-only judges given video summaries miss temporal errors. These two results lead us to use video-ingesting judges with per-event rubrics. AuroraCap's VDCScore breaks references into question-answer checks, and AutoEval-Video uses must-have and must-not-have rules. Both are flat checks, and we use them as comparison points.

One paper constrains the design. Is Your Video Language Model a Reliable Judge? finds that open video models are "easy graders" that cluster scores at 4 to 5 and miss non-existent actions. It also finds that ensembling them can do worse than GPT-4o alone, with agreement scores between 2.72% and 42.53%. This rules out committee judging. It also raises a direct risk for our pipeline, namely that the judge misses inherited errors precisely because they are self-consistent. We therefore measure judge snowballing directly instead of assuming it away.

Research Program

The first component is the measurement protocol. Each video is split into its annotated events. The captioner describes event $k$ given the clip and its own descriptions of events $1$ to $k-1$. This covers streaming captioners and general video models prompted event by event: Qwen2.5-VL-7B, LLaVA-Next-Video, MiniCPM-V-2.6, and Gemini-2.0-Flash. A judge sees the event's clip, the reference description, and a rubric generated for that instance in the style of VideoJudge, and marks unsupported claims as atomic statements. For each erroneous event the judge writes the smallest edit that makes the description correct. We re-generate the remaining events from both the original and repaired prefixes, several times each. The judge then matches surviving error claims across runs by meaning, not wording, so that a reworded copy of the same false claim still counts.

The second component checks the measurement with known causes. Using the "adversarial trajectory" idea of MM-Snowball, we take ground-truth prefixes and inject a single controlled error into event $k$: a wrong object, a swapped actor role, or a reversed action. Every downstream error that appears is then known to be caused by the injection. This produces labels for scoring attribution methods and gives an induced-error rate for each model and each error type.

The third component audits the judge. We plant matched pairs of errors at event $j$ that either agree with an earlier planted hallucination or stand alone. We then measure the judge's detection rate on each, for GPT-4o and VideoJudge-7B, with the judge reading either the full narrative or only event $j$ in isolation. If detection drops for the errors that agree with an earlier hallucination, judges snowball too. In that case the protocol must judge each event against only its own clip, and we will report this as a finding.

Baselines

The naive reference is free-running versus ground-truth-prefix generation. We run each captioner once with its own history and once with reference history, then compare aggregate error rates from the same judge. This measures the overall exposure-bias gap at about twice the cost of plain evaluation. Any per-error method must at least recover this gap when its attributions are summed.

The surface-metric baseline is CIDEr and BLEURT computed per event. On paragraph summaries, DeVAn reports BLEURT reaching 67% alignment with human preferences, while CIDEr fails to correlate. We include both to show that neither changes when inherited errors are repaired. That behavior would show they are blind to the distinction, not just noisy.

The strongest flat existing method is ARGUS-style GPT-4o entailment. We run it on the same captions with the same judge model. On ArgusBench, the best reported hallucination cost is 41% ArgusCost-H for Gemini-2.0-Flash. We will reproduce scores on that scale before adding attribution.

The strongest attribution competitor is an AgentEval-style judge. GPT-4o reads the event-by-event trace with the video and labels each error as root or propagated, under a prompt and few-shot budget equal to ours. AgentEval reports 72% root-cause accuracy in text workflows. No reported result was found on Lacuna for any attribution method on video captions. We also include a "first error is root, later agreeing errors are propagated" heuristic. Fairness means identical captions, identical judge models, and comparable judge-call budgets. Our method's extra cost is captioner re-runs, which we report separately.

We count it as beating these baselines if counterfactual repair attributes injected-error consequences with clearly higher accuracy than judged attribution, with bootstrap confidence intervals that do not overlap, and agrees better with expert labels on naturally occurring errors. The key ablation removes the placebo re-run and uses the logged original output instead. This isolates how much of the measured "inheritance" is sampling noise. A second ablation replaces the minimal correction with a ground-truth sentence, to test whether rewording alone shifts later outputs.

Why not just…?

Why not just measure the free-running versus ground-truth-prefix gap? This is the real threat. If the only goal is a single exposure-bias number per model, it is cheaper and probably sufficient. Our method adds two things the gap cannot give. The first is which errors and which error types propagate: swapped actor roles may spread while attribute errors stay local, which matters for the role-reversal failures described in The 8th AI City Challenge. The second is a repair that changes a single event, whereas a full ground-truth prefix also changes style and length. If per-type inheritance turns out to be nearly uniform across models, we will say the aggregate gap is enough.

Why not just ask a strong judge which errors are propagated? A judge reading text sees correlation: two related false claims. It cannot tell whether the second was caused by the first or misperceived independently from the same ambiguous frames. The injected-error experiment tests this directly against known causes. The judge audit tests whether the judge's own consistency bias makes it miss exactly the errors that matter.

Why not just use ARGUS or VDCScore and look at error positions? Both produce error lists that already exist, and the first-error heuristic built on them is one of our baselines. If it matches counterfactual attribution on injected errors, the intervention is unnecessary and we will report that. We expect it to fail when a model recovers after an error, the V-shaped recovery MM-Snowball observes, and when separate local errors happen to come after the first one.

Why not a text-only judge over frame captions to save cost? VideoJudge reports that text summaries miss temporal errors. Our judge must verify each repaired event against its clip, so we keep video-ingesting judges and use a text-only judge only as a cost comparison.

Proposed Experiments

We use YouCook2 and ActivityNet Captions, whose ordered events make propagation plausible; recipe steps in YouCook2 depend strongly on earlier ones. We use ArgusBench's 500 human-captioned videos for the flat-metric anchor, and VDC for long descriptions. On about 300 videos per dataset, we run injected-error trials across three error types and several injection positions. This yields attribution accuracy for every method against known causes and induced-error curves for each captioner. On naturally occurring errors, experts label a subset of about 150 narratives for root versus propagated, using the human-ceiling protocol of AgentEval.

The decisive checks are four. First, placebo re-runs must show that sampling noise accounts for much less than the measured inheritance. Second, counterfactual attribution must beat judged attribution on injected errors. Third, the inheritance rate should vary across models even when their ArgusCost-H scores are similar. That would show flat scores hide a real difference. Fourth, the judge audit determines whether the evaluator can be trusted to read whole narratives. If the inheritance rate is near zero for current models, the conclusion is that event-by-event captioners do not snowball the way image dialogue models do. That is a useful negative result, and the injected-error benchmark remains a stress test for future streaming captioners.