Frame Citations That Survive Deletion: Auditing What Sparse Video Samplers Claim to Have Seen

Video models that answer questions about long recordings rarely look at every frame. They pick a few percent of frames, by uniform spacing or by scoring frames for relevance to the question, and answer from those. In surgery, the selected frames are tempting to present to a clinician as the explanation: "the model answered this because of these five moments." This proposal tests whether that presentation is honest. A cited frame set is honest if two things hold. The answer can be reproduced from the cited frames alone, which we call sufficiency. The answer changes or becomes uncertain when those moments are removed, which we call necessity. We will build a sampler that selects frames to pass both checks, and we will record each check in a replayable audit record so a third party can repeat it.

The reason this is not routine is temporal redundancy, the property that makes sparse sampling work in the first place. Consecutive frames are near-copies. If a model cites frame 1,204, deleting that one frame proves nothing, because frames 1,200 to 1,210 carry the same content. Frame-level deletion tests therefore make almost any citation look unnecessary. They also let decorative citations pass, since the answer survives deletion through the neighboring frames either way. Our central claim is that necessity in video must be tested over redundancy clusters, meaning stretches of near-duplicate frames, rather than over single frames. A sampler that optimizes for cluster-level sufficiency and necessity will produce citations that are much more often real evidence than those of query-relevance samplers, at similar accuracy and frame budget.

Where this sits

Adaptive keyframe selection is mature for general video. FOCUS treats temporal clips as bandit arms, meaning candidate options it spends a limited scoring budget on to find the most relevant ones. It scores 1.6% of frames and beats uniform sampling by 11.9% on LongVideoBench videos over 20 minutes. Adaptive Keyframe Sampling balances query relevance against temporal coverage. VideoEspresso shows that about 2% "core frames" can improve reasoning. These papers supply our search machinery and our strongest baselines. All three select frames for answer accuracy. None asks whether the selected frames are what the answer actually depends on, so none of them evaluates its selections as explanations.

On the surgical side, LoViT classifies phases from 1 fps video with sparse attention but offers no evidence for individual decisions. LLaVA-Surg answers free-form surgical questions. Its authors report confident hallucinations and no link from answers back to frames, which is the gap that motivates this work.

Two lines of grounding work supply our verification logic, and both stop at static inputs. Self-Verification makes a model quote exact evidence spans from clinical notes and prune unsupported outputs. LedgerMind logs tool outputs into a structured evidence ledger, limits repairs to typed operations (drop, retry, abstain), and names "phantom grounding," which is citing sources that do not contain the claimed fact. Text spans and image crops have no redundancy problem: one quote is one piece of evidence. Video does have this problem, and that is what we contribute.

The integrity literature sets the standard we hold citations to. CreditMap shows that granular, append-only provenance records answer 94% of audit queries, against 18% for coarse role taxonomies. The many published author and data-credit corrections, such as Weissler et al. and Xu et al., show what happens when attribution is patched after the fact instead of recorded at the time. We take one narrow lesson from this work: a citation is only worth something if someone else can check it. The ledger itself is ordinary engineering. The scientific content is the checks it records.

Core mechanism

First, we partition each video into redundancy clusters. We merge adjacent frames whose embeddings, from the backbone's own visual encoder, stay within a similarity threshold $\tau$. The threshold is set on held-out data so that within-cluster frame swaps leave the backbone's answer unchanged at least 95% of the time.

Next, a FOCUS-style coarse-to-fine bandit proposes a small cited set $C$ of clusters. We keep FOCUS's empirical Bernstein confidence schedule, which decides how much budget each clip gets. What we change is the score. Instead of query relevance alone, the score is a gap in answer confidence, written below in plain notation:

score(C) = p(a | C) − max over C' of p(a | V \ C ∪ C')

Here $a$ is the answer the backbone gives from $C$. The first term, $p(a \mid C)$, is the backbone's probability of $a$ when shown only the cited clusters, and it measures sufficiency. In the second term, $V \setminus C$ means the video with the cited clusters removed, and $C'$ is a same-size set of uncited clusters drawn from elsewhere in the video and inserted in their place. This keeps the frame count and temporal coverage fixed. The maximum over a few such draws measures how well the answer survives without the cited evidence, so a large gap means the cited clusters are necessary.

If sufficiency fails, the sampler adds clusters. If necessity fails, it flags the citation as decorative and either searches again or abstains. This mirrors LedgerMind's restricted repair operations, with no open-ended free-text self-correction.

Every query produces a record containing:

  • the video hash and the cluster boundaries;
  • hashes of the cited frames;
  • the replacement draws and the backbone and prompt version;
  • both probabilities.

An auditor holding the same video can replay both tests exactly. Only hashes and timestamps are stored, never pixels. This sidesteps the privacy tension that ProtoMedAgent raises for evidence stores containing raw patient data.

Research program

The work has three parts.

Measurement. We first measure how bad current citations are. We run uniform sampling, Adaptive Keyframe Sampling, and FOCUS on surgical and general long video, ask each backbone to cite the frames it used, and apply frame-level and cluster-level tests to those citations. The prediction is that frame-level necessity looks spuriously low for every method, and cluster-level testing exposes large differences between methods.

The sampler. Second, we build the sufficiency-and-necessity sampler described above and characterize the price of faithfulness. The cost has three parts:

  • lost accuracy;
  • extra backbone calls, roughly three per candidate set;
  • added frames.

Transient events. Third, we study short events, a known weakness of coarse sampling (noted in Temporal Segment Networks and in the FOCUS limitations). A short event forms its own small cluster, so cluster-level necessity should reward finding it rather than penalize it as an outlier.

Baselines

Uniform sampling. This is the naive reference: frames evenly spaced at the same budget, with the backbone asked to cite timestamps. It is the floor for both accuracy and citation faithfulness, and A system for TRECVID MED by MCIS shows it is a strong classification baseline.

FOCUS with post-hoc citation. This is the strongest existing competitor. It is the best-reported adaptive selector, and its selections are the most natural thing to show a clinician as evidence. The only reported figure on Lacuna is the relative +11.9% over uniform on LongVideoBench videos over 20 minutes. No absolute accuracy for FOCUS was found on Lacuna, so we will reproduce both FOCUS and uniform ourselves.

Adaptive Keyframe Sampling. We include it as a second relevance-based selector.

Self-verification on frames. This applies the Self-Verification pipeline to frames: extract, check for omissions, ground evidence, prune.

Fairness and the bar for beating these. All methods share the same backbones and the same per-query budget of frames shown and backbone calls. Baselines that use fewer calls get the surplus as extra frames. Thresholds are tuned on the same validation split with the same number of trials. Beating FOCUS means two things together. Cluster-level necessity and sufficiency rates must rise by at least 15 points. Answer accuracy must be within 2 points of FOCUS, or above it.

Domain reference. On Surg-QA we report LLaVA-Surg's published GPT score of 2.45 and Accuracy@all of 0.308.

Phase-recognition context. On Cholec80 we report LoViT's 91.50% video-level accuracy and 74.15% Jaccard, and on AutoLaparo its 81.43% accuracy. These numbers are context only. LoViT sees all 1 fps frames, so it sets a ceiling for sparse methods rather than a competitor.

Ablations. The key ablation removes the redundancy clusters and runs the identical sampler with frame-level deletion. If this matches the full method, our central claim is wrong. A second ablation drops the necessity term and keeps sufficiency only.

Why not just…?

Why not just use FOCUS and ask the model to cite its frames? This is the default and the main threat. Nothing in the file shows that relevance-selected frames fail as evidence. The measurement study settles it directly. If FOCUS citations already pass cluster-level tests at high rates, the new sampler is unnecessary, and the contribution shrinks to the evaluation protocol. We would report that outcome as the finding.

Why not just run self-verification over frames? The Self-Verification result is strong: 93% of its evidence spans overlap with expert spans. Its pruning step asks the model whether evidence supports the answer. It never removes the evidence and observes the result. LedgerMind's phantom-grounding analysis suggests that self-judged support is exactly where unfaithful citations slip through. This approach is a real competitor, and it is run as a baseline at matched call budget.

Why not just use the model's attention over uniformly sampled frames as the explanation? Attention weights over redundant frames spread across near-duplicates, and they are not tested against what happens when frames are removed. Our cluster-level deletion test is the same measure we apply to every method, so attention-based citations get scored on equal terms. We include attention-top-k citations as an extra arm in the measurement study.

Why not just hash and log every frame? Logging is cheap: CreditMap reports under 0.1 ms per event. A complete log shows what the model was shown, not what it relied on. The record matters only because it holds replayable sufficiency and necessity results. We do not claim the ledger as a contribution.

Proposed experiments

Datasets. We use Cholec80 and AutoLaparo with question templates built from phase annotations, such as phase order, whether a phase is present, and when a transition happens. The annotated boundaries give approximate ground-truth evidence windows. We also use Surg-QA for free-form questions and LongVideoBench to check generality.

Insertion test. The decisive test inserts a 1 to 3 second clip from a different procedure or video at a random point and asks a question only that clip can answer. The ground-truth evidence location is then exact.

Metrics. We measure:

  • answer accuracy;
  • frames scored and frames shown;
  • backbone calls;
  • sufficiency rate and cluster-level necessity rate;
  • phantom-citation rate, meaning cited clusters with no overlap with ground-truth evidence;
  • temporal overlap (IoU) between cited and true evidence on inserted clips.

Backbones. We use LLaVA-Surg and Qwen2.5-VL, the open models used in the cited work.

Decisive checks. The central claim needs both of these:

  • On inserted clips, the full method's citations overlap the true clip clearly more often than FOCUS's.
  • The cluster-to-frame ablation gap shows up on surgical video, which has long static stretches, much more than on general video.

If cluster-level and frame-level necessity rank methods identically, then redundancy does not distort faithfulness testing, and the proposal's premise fails.