Skip the Frame or Call the Committee? Triage for Ensemble Video Recognition

When a video classifier is unsure about a frame, there are two very different reasons it might be unsure. The frame may be useless: blurred, occluded, or showing the player from behind, so that no model could read it. Or the frame may be clean but genuinely hard, so that a second, differently trained model could plausibly get it right. Existing systems treat these two cases the same way. Frame-filtering methods drop uncertain frames. Inference-efficient ensembles, which run a cheap model first and call more models only when needed, spend more compute on uncertain inputs. This proposal builds a per-frame triage rule that uses the cheap first model's own output to make a three-way decision: accept the frame, skip it and move to the next frame, or expand to more ensemble members on this frame. The rule works along two axes at once, time (which frames) and ensemble width (how many models).

The claim is testable and could turn out false. Among frames on which a cheap model is uncertain, a substantial fraction are "unrescuable," meaning every ensemble member fails on them. We claim these frames can be told apart from "rescuable" ones before any extra model runs. If both parts hold, routing unrescuable frames to "skip" and rescuable ones to "expand" should give a better accuracy–compute trade-off than either filtering alone or ensemble halting alone. The natural test bed is soccer jersey-number recognition. There, Balaji et al. report that the number is visible in only a small minority of frames: their filter discards 87.65% of frames. That makes the garbage-versus-hard distinction the central problem of the task rather than a corner case.

Core Claim

We use evidential outputs, in which a model predicts parameters of a Dirichlet distribution over classes instead of a single softmax. For class evidence $e_k \ge 0$ over $K$ classes, the Dirichlet parameters are $\alpha_k = e_k + 1$ and the total strength is $S = \sum_k \alpha_k$. This yields two separate signals. The first is vacuity, $u = K/S$, which is high when the model sees little evidence for anything. The second is dissonance, which is high when substantial evidence is split between competing classes. Our hypothesis is that vacuity on the first ensemble member predicts unrescuable frames, and that dissonance, or middling confidence with high evidence, predicts rescuable ones. A small learned gate reads these signals together with cheap image statistics and emits accept, skip, or expand. Evidence from accepted frames is summed across the tracklet, and the tracklet halts once its pooled posterior is confident.

What is new is the skip branch inside an ensemble cascade, and the empirical question of whether it can be predicted. IRENE chains base classifiers and halts early when confident. It has no notion that an input might be worthless, and it runs only on static CIFAR images. AdaFrame decides which frame to look at next, but with a single backbone and a learned utility score rather than any cross-model signal. Grad uses Dirichlet evidence to abstain on ambiguous jersey frames and reports a 5–6% gain over softmax. That supplies our uncertainty machinery, but it involves one model and never asks whether a second model would help. Taken together, these papers show that the same signal, low confidence, is used for opposite actions in the two literatures. We aim to resolve that tension with data.

Research Program

The first stage is a headroom study, run before any gating is built, because the whole proposal depends on it. On SoccerNet tracklets we train three diverse jersey-number recognizers: a small convolutional network as the cheap first member, a larger convolutional network, and a vision transformer. Each uses an evidential head in the style of Grad. Every frame is then labeled by outcome. A frame is easy if the first member is correct and confident. It is rescuable if the first member is wrong or unconfident but a later member or the combined ensemble is correct. It is unrescuable if all members fail. We report these fractions and how rescuable frames are distributed across tracklets. If rescuable frames are rare once unrescuable ones are removed, multi-model inference has little to offer on this task. In that case the right conclusion is that a single filtered model suffices, and we will report that finding.

The second stage trains the gate. It is a small multilayer network whose inputs are the first member's Dirichlet parameters, its vacuity and dissonance, and inexpensive frame statistics (a sharpness score and crop size). It is trained with cross-entropy against the outcome labels from stage one, on a held-out split so that the members' training-set overconfidence does not leak into the labels. Later members are fine-tuned with a specialization loss adapted from IRENE's ranking loss, restricted to rescuable frames. We also add a confident-oracle term from Confident Multiple Choice Learning, which trains non-specialist models to output high-entropy predictions outside their domain. Here it pushes later members toward high vacuity on unrescuable frames, so their mistakes do not contaminate the pooled evidence.

The third stage is the temporal policy. By default, frames are visited in order with the cheap member, so no separate full-video pre-scan is needed; the first member itself acts as the scanner. The tracklet halts when pooled evidence reaches a confidence threshold. Compute is measured as the total floating-point operations per tracklet, including every first-member pass and every gate call. We sweep the halting threshold and the gate's decision thresholds to trace accuracy-versus-compute curves.

To test generality beyond a sports setting, we port the method to ActivityNet v1.3 and FCVID. These are untrimmed action benchmarks where AdaFrame reports its results. The ensemble there consists of three action backbones, and triage replaces AdaFrame's single-model glimpses.

Baselines

The naive reference processes every frame of the tracklet with the single strongest member and averages its predictions. It shows what filtering and ensembling each add. A second reference, uniform stride subsampling with the same single model, is included because a reviewer will ask for it. Balaji et al. found that unfiltered temporal modeling reached only 30.89% against 68.53% with keyframe filtering, so we expect it to be weak.

The strongest existing method on SoccerNet is the single-stage evidential recognizer of Grad, which reports 85.62% tracklet accuracy on the SoccerNet Challenge, against 79.31% for earlier methods. We re-implement it on the same splits and use it as the "filter, single model" operating point. It is the real competitor because it already exploits uncertainty to discard frames and does so at single-model cost. To beat it, our method must match its accuracy at no more than its per-tracklet compute, or exceed it by a margin beyond seed variance at a stated compute multiple. We will report both conditions. Every model shares the same crops, data augmentation, and training budget, and every gating threshold is tuned on the same validation split with the same number of sweep points.

The strongest ensemble baseline is IRENE ported to frames, reproducing Li et al.: the same three members, the same ranking loss, and the RNN-based halting selector, with per-frame predictions pooled per tracklet. Its reported result is on CIFAR-10: a 56% cost reduction with 95.81% accuracy against 95.62% for the full ensemble. No video result for IRENE was found on Lacuna. This baseline is also our key ablation, because it is our system with the skip branch removed. A second ablation removes the expand branch, leaving triage with skip only, which is an evidential filter over the cheap model. A third replaces the learned gate with two fixed thresholds: skip if vacuity exceeds $\tau_u$, accept if confidence exceeds $\tau_c$, and expand otherwise.

On ActivityNet the reference is AdaFrame, which reports 71.5% mean average precision using an average of 8.65 frames, against 70.4% for an LSTM at the same frame count. We will compare at matched compute rather than matched frame count, since our frames can cost more than one model pass.

Why not just…?

Why not just filter frames and run one distilled model? This is the most serious threat. Collaborative learning (Guo et al.) and DMCL-style distillation (Garcia et al.) put ensemble knowledge into one network at no extra inference cost, and Grad's filter already removes junk frames. If the stage-one headroom study shows few rescuable frames, or if a distilled student recovers most of the ensemble's gain on filtered frames, the "expand" branch is not worth having. We therefore include a single student distilled from the three-member ensemble and run under the same evidential filter as an explicit baseline. The proposal succeeds only if triage beats it on the accuracy–compute curve.

Why not just use a quality filter in front of an IRENE cascade? A separate quality network, or heuristics such as those of Balaji et al., could remove garbage frames before IRENE runs. Our argument is that the first member's evidence is a free and task-specific quality signal. Hand-set region-of-interest and color-histogram thresholds fail when broadcast conditions change, and a separate quality network adds a pass on every frame. We test this directly by comparing a blind image-quality filter followed by IRENE against the learned gate, both at matched compute.

Why not just threshold the first model's vacuity and confidence with no learned gate? It may work as well as the learned gate, and the two-threshold ablation exists for exactly this reason. If it matches the learned gate, we will present the simple rule as the method. The contribution would then be the demonstration that vacuity separates unrescuable from rescuable frames, together with the measured compute savings. A softmax-only version of the same rule is also run, because Grad and Lee et al. both report that softmax is overconfident on corrupted or out-of-domain inputs. We expect this to hurt the skip decision in particular.

Proposed Experiments and Decisive Checks

The first decisive check is the headroom table from stage one. It gives the fractions of easy, rescuable, and unrescuable frames, together with the tracklet accuracy of the full three-member ensemble compared with the best single member on filtered frames. If the ensemble's gain over the best single member is within seed noise, we stop and report a negative result.

The second check is separability. We measure how well the gate, and vacuity alone, discriminate rescuable from unrescuable frames among those the first member finds uncertain, reported as area under the ROC curve. The skip branch is justified only if this discrimination clearly exceeds that of softmax entropy.

The third check is the main comparison. We plot accuracy-versus-compute curves on the SoccerNet Challenge split and on the second test set with a domain gap described by Balaji et al.. The curves cover triage, IRENE-on-frames, the distilled single student under filtering, the Grad re-implementation, and the three ablations, all with three seeds. Success means triage lies above IRENE-on-frames at every compute level: the skip branch saves compute that IRENE wastes on junk frames. It also means triage lies above the filtered single models at some compute level: the expand branch buys accuracy that distillation does not.

The final check is transfer. We repeat the curve comparison against AdaFrame on ActivityNet v1.3 and FCVID. In trimmed or cleaner footage, where unrescuable frames may be rarer, we expect the skip branch to matter less. Reporting where the advantage shrinks tells practitioners when triage is worth building.