Auditing Scene Bias in Video Models with Abort Attacks from Fair Coin Tossing

Video action classifiers often decide from the first frame. They see a pool and answer "swimming" before any motion has been shown, and they stay wrong when the swimmer turns out to be doing something else. This note proposes a way to detect that behaviour using only the model's own predictions over time, with no labels, no background-swapped test sets, and no paired empty-scene footage.

The idea comes from fair coin tossing in cryptography. Two parties build a random bit over several rounds, and a cheater may quit at the round most favourable to them. That literature reasons about the running probability of the final outcome as a martingale: a sequence of predictions in which today's value equals the expected value of tomorrow's. Our proposal treats a video model's frame-by-frame class probabilities the same way. A model that uses evidence honestly, including legitimate context such as a basketball hoop, produces predictions that behave like a martingale. On average they neither drift up nor down as frames arrive. A model that is overconfident because of the scene makes early predictions that the later motion evidence reliably erodes. That erosion is a measurable drift, and a "quitting adversary" who picks the best moment to stop the video can extract it.

Core Claim

Take a causal video model, meaning one that emits a class distribution $q_t$ after seeing frames $1,\dots,t$ of a $T$-frame clip. If $q_t$ is the correct posterior given the prefix, it satisfies

$$\mathbb{E}\left[\,q_{t+1} \mid \text{frames}_{1..t}\,\right] = q_t ,$$

so by the optional stopping theorem no stopping rule $\tau$ can make $\mathbb{E}[q_\tau(c) - q_T(c)]$ nonzero for any class $c$. We define the abort gain of a model on a video distribution as the largest such expected gap achievable by a stopping rule that watches the predictions:

$$G = \max_{\tau \in \mathcal{S}} \; \mathbb{E}\left[\, q_\tau(\hat c_1) - q_T(\hat c_1) \,\right],$$

where $\hat c_1$ is the model's top class after the first frame and $\mathcal{S}$ is a family of threshold-based stopping rules.

The central empirical claim is that $G$ is near zero on the training distribution even for models that lean on context, because context there is predictive and consistent. Under a scene shift, $G$ becomes large and positive. This yields a label-free, per-video and per-dataset signal of scene reliance. Because it penalizes only context that the rest of the video contradicts, it separates spurious context from legitimate context.

That separation targets the open problem the file attributes to ALBAR: blanket suppression of static cues discards useful context. The measurement also needs no paired empty-background clips of the kind PanAf-FGBG relies on, and no synthetic background swaps of the kind SCUBA uses.

Where This Sits in Prior Work

The cryptographic line provides the method.

Cleve's bound and the matching protocol of An Optimally Fair Coin Toss show that a quitting adversary's power is governed by jumps in the running expectation of the outcome. Estimating Gaps in Martingales and Applications to Coin-Tossing makes those jumps the object of analysis.

Two further papers supply the attacker we adapt, one that estimates future conditional expectations and then picks its moment to quit. On the Black-Box Complexity of Optimally-Fair Coin Tossing builds such an attacker by learning shared dependencies. Can Optimally-Fair Coin Tossing Be Based on One-Way Functions? builds one by simulating plausible futures.

We change the adversary's role. In cryptography, gaps are the protocol designer's enemy, and some gaps are unavoidable, with a lower bound of order $1/r$ over $r$ rounds. In video, jumps are fine because evidence genuinely arrives in bursts. The fault is systematic drift. So we borrow the attacker as an auditor, not the fairness bound as a target.

The random-oracle literature plays a narrower role: it names the assumption we avoid. In cryptography, a random oracle is an idealized hash that returns fresh uniform randomness. Proofs that assume one can fail once real code replaces the ideal hash (Non Observability in the Random Oracle Model). Fast White-Box Adversarial Streaming Without a Random Oracle shows that such guarantees can be rebuilt against concrete functions.

Decoupling methods such as Dual Contrastive Learning for Spatio-temporal Representation and the two-stream design of CAST implicitly treat the encoder as if it made static and dynamic features independent. They offer no check that the deployed encoder actually does so. Our audit makes no independence assumption. It tests the concrete trained network against its own future predictions. We use this analogy only as a design constraint and claim no formal cryptographic guarantee.

The vision line motivates the gap and constrains the design.

ALBAR trains models to be uncertain on motionless clips and reports that naive versions collapse into a "trivial solution" that needs entropy and gradient regularizers. It also flags background leakage in earlier benchmarks. PanAf-FGBG shows that background-only models reach about 65% of full-model performance. It improves out-of-distribution results by subtracting background embeddings, but that requires paired empty clips. CATER removes scene cues entirely and exposes weak temporal reasoning.

None of these measures when a model commits to its answer, or whether that commitment is later contradicted.

Research Program

Building the prefix readout. The first step turns standard backbones into causal predictors. We attach a lightweight prefix head to frozen or fine-tuned models: I3D-style convolutional networks, an MViT transformer, and CAST. The head is trained with cross-entropy on labels at randomly sampled prefix lengths. Proper scoring losses make the population minimizer the prefix posterior, so in distribution the readout should be approximately a martingale, and any residual in-distribution drift serves as the null reference.

Defining the auditor. The second step specifies the stopping-rule family $\mathcal{S}$. Rules stop at the first frame where $q_t(\hat c_1)$ crosses a threshold, or where its change crosses a threshold. The per-video score is the realized gap $q_\tau(\hat c_1) - q_T(\hat c_1)$. We standardize it against the held-out in-distribution null. The design choice that matters is that the adaptive rule exploits when drift occurs, not just its endpoint size.

Test-time correction. The third step, secondary and only if the audit works, uses the audit at test time. When a video's score exceeds the null, we down-weight early-prefix evidence in the final decision. We then check whether this recovers accuracy without retraining.

Baselines

The naive reference is the unmodified backbone evaluated on full clips, with its maximum softmax probability used as the per-video suspicion score. If abort gain does not beat plain confidence at flagging scene-driven errors, the proposal fails.

The strongest measurement competitor is fixed-prefix disagreement: the divergence between the first-frame prediction $q_1$ and the full-clip prediction $q_T$. It is the obvious simplification of our score. It also serves as the key ablation, since it removes the adaptive stopping rule and the in-distribution null calibration while keeping the same prefix head. A closely related competitor is the model's confidence on a motionless version of the clip, in the style of SCUFO. This is ALBAR's diagnostic, and it is the closest existing label-free probe of scene reliance.

The strongest mitigation competitor is ALBAR itself. It reports a 12% absolute increase in Contrasted Accuracy on HMDB51 and the best results on SCUBA and SCUFO across its tested datasets. The file gives no per-benchmark SCUBA or SCUFO numbers: no reported result found on Lacuna for ALBAR's absolute SCUBA/SCUFO accuracies. On PanAf-FGBG, the strongest baseline is latent background compensation, which reports +5.42 mAP out of distribution for convolutional models and +3.75 mAP for MViT.

For fairness, every method uses the same backbone checkpoints, training data, and frame sampling. Each gets the same hyperparameter search budget, tuned on the same in-distribution validation split, and nothing is tuned on shifted test sets. Beating ALBAR at detection means higher AUROC for flagging misclassified shifted videos at matched compute. At mitigation, parity with ALBAR on SCUBA and ARAS combined with strictly better in-distribution accuracy on context-predictive classes would count as a meaningful win. We would not claim a win on SCUBA alone.

Why not just…?

Why not just use ALBAR's motionless-clip test or train with it? That probe asks whether the model is confident without motion. It cannot tell a hoop that is legitimately predictive from a pool that misleads, and ALBAR's own training needed extra regularizers to avoid degenerate solutions. Our experiments test this directly. On Kinetics400 classes where the background-only baseline is strong and correct, we measure whether each method lowers in-distribution accuracy. The prediction is that ALBAR-style suppression does and martingale-consistency auditing does not. If both behave alike, that part of the claim is falsified.

Why not just compare the first-frame and full-clip predictions? This is the real threat, because it is simpler and uses the same ingredients. Its weakness is that first-to-last disagreement is also large in distribution whenever evidence legitimately arrives late, so its false-positive rate should be higher. The ablation is decisive here. If fixed-prefix disagreement matches abort gain in AUROC on SCUBA, ARAS, and PanAf $D_{\text{disjoint}}$, we will report that the adaptive attacker adds nothing. The contribution would then shrink to the claim that prefix-posterior drift, measured against an in-distribution null, is the right label-free scene-bias statistic.

Why not just subtract the background, as PanAf-FGBG does? Subtraction needs an empty clip from the same camera, which web video such as Kinetics or UCF101 lacks. PanAf also reports that performance falls sharply as the share of background-only frames grows. We still run it on PanAf, where pairs exist, as an upper-reference baseline.

Why not just use confidence-based out-of-distribution detection? A scene-biased model is often confidently wrong on the full clip, so plain confidence should miss exactly the cases we care about. The naive baseline tests this.

Proposed Experiments

Detection. Using prefix heads on the three backbones, we score every test video on SCUBA (background swaps), ARAS (rare backgrounds), and PanAf-FGBG $D_{\text{disjoint}}$ against $D_{\text{overlap}}$. We report AUROC for flagging misclassified videos, compared against maximum softmax, motionless-clip confidence, and fixed-prefix disagreement.

Dataset-level validation. Across PanAf behaviours and Kinetics classes, we correlate each class's average abort gain with the background-only gap from PanAf-style background training. A positive rank correlation would show that the label-free score tracks the labelled measure of scene reliance.

Negative control. On CATER, where there is no scene to exploit, abort gain should stay near the null under all splits. This holds even though accuracy is poor, with reported Snitch Localization Top-1 of about 26% for 3D models and 46% with an LSTM. A large gain there would mean the statistic is picking up temporal-reasoning failure rather than scene bias.

Mitigation. We apply test-time correction and compare against ALBAR and latent compensation on SCUBA, SCUFO, and ARAS using Contrasted Accuracy and top-1. On PanAf we compare out-of-distribution mAP against the reported +5.42 and +3.75 gains. For every method we also report in-distribution accuracy on Kinetics400, UCF101, and HMDB51.

The decisive checks are three. The auditor must beat fixed-prefix disagreement on detection. It must not hurt context-predictive classes where suppression methods do. It must stay silent on CATER. Failing the first reduces the work to a simpler statistic, and failing the second removes the reason to prefer it over ALBAR.