Rare Video Actions Lean on Scene-Summary Tokens: Diagnosing and Breaking the Shortcut

When a video model sees a rare action only a handful of times, the easiest way to fit those examples is to memorize where they happened rather than what happened. This proposal asks whether that shortcut has a precise address inside a Vision Transformer, a model that cuts each frame into small square patches, turns every patch into a token, and lets all tokens attend to one another. Recent work on image-language models found that these encoders spontaneously produce a tiny set of "attention sink" tokens, roughly 0.5% of all visual tokens with unusually large feature norms. Those tokens carry a global summary of the image, such as scene category and lighting, while the remaining tokens carry local detail (To Sink or Not to Sink). Our hypothesis is that under class imbalance, rare action classes are predicted disproportionately from these scene-summary tokens, and frequent classes from local hand-object tokens. If so, the fix is cheap. During fine-tuning, withhold the sink tokens from rare-class examples, with a probability tied to class rarity, so the model has to find the evidence in the local patches.

The problem is sharpest because video-text data is scarce. Since paired video-text corpora are small, the standard recipe keeps a ViT pretrained on image-text pairs and adds only light temporal machinery on top (BT-Adapter), or stretches scarce human narrations with machine-generated ones (LaViLa). Both routes inherit static-image token structure, sinks included, and both fit rare actions from very little data. Egocentric kitchen video makes the stakes concrete. In EPIC-KITCHENS-100, strong models reach about 38% top-1 action accuracy but often fall below 15% on tail classes, and performance also drops across environmental shifts (Rescaling Egocentric Vision). These are exactly the conditions under which a scene shortcut would both help on the training distribution and break off it.

Core Claim

The proposal makes two claims, each testable on its own. The first is diagnostic. In ViT-based video models fine-tuned on long-tailed data, the drop in true-class confidence when sink tokens are masked at test time grows as class frequency falls. Tail classes depend more on the global scene summary than head classes do. The second is interventional. Masking sinks during training, in proportion to class rarity, improves tail accuracy most on test clips recorded in environments that differ from the training clips. It should cost almost nothing on head classes and add negligible compute.

What is new is locating the long-tail context shortcut in a specific, identifiable token population, and intervening on that population. Prior work fights the same pathology at the level of pixels, frames, or pooled features.

Relation to Prior Work

Several papers motivate the proposal. The Majority Can Help the Minority (CMO) shows that tail classes lack contextual diversity and learn backgrounds as shortcuts. It fixes this by pasting minority foregrounds onto majority backgrounds, raising ImageNet-LT few-shot accuracy from 5.8% to 20.5%. That method works on static images and introduces paste artifacts. We target the same pathology without synthesizing pixels.

To Sink or Not to Sink supplies the token-identification method, which finds sinks by feature norm. It uses sinks constructively, routing them through a separate projector for image question answering. That finding also constrains us: it shows sinks carry useful semantics. We therefore do not delete sinks at inference, and we leave head-class training untouched. Removing sinks entirely is tested below as a failure case.

The video long-tail literature constrains the design space. VideoLT argues that oversampling whole videos overfits to background frames. Its FrameStack method resamples at the frame level. Selective Volume Mixup argues that action evidence sits in sub-frame patches and selects patches by cross-attention. However, it was never run on imbalanced data and costs a 52% drop in training throughput. Use Your Head (LMR), MOVE, and MEID borrow or reshape head-class features to enrich the tail. All three work at the level of pooled clip or frame features, mostly on frozen CNN features. None of these methods asks which tokens a transformer uses for tail decisions.

Our proposal moves the intervention down to the patch-token level, consistent with the SV-Mix argument, but needs no second video and no extra forward pass. Decoupled and Patch-based Contrastive Learning transfers local patch primitives from head to tail on images. It is complementary to our approach: it adds local evidence, while we remove the global crutch.

Research Program

The first stage is measurement. We will take a CLIP-initialized video ViT and fine-tune it under two regimes. One is a divided space-time model in the style of BT-Adapter. The other is a LaViLa-style dual encoder pretrained on Ego4D narrations, with and without synthetic narrations. In each frame we identify sinks as the tokens with the largest feature norms, using the roughly 0.5% rate reported for images as a starting point. We will also check whether video sinks are per-frame or persist across frames.

For every test clip we compute a sink-reliance score. This is the drop in true-class log-probability when sink tokens are masked as attention keys at every layer. We compare it with the drop from masking an equal number of random non-sink tokens. We then plot reliance against training-class frequency. A flat curve falsifies the premise and ends the project at stage one, which is still a reportable negative result. We will also check what the sinks encode by training a linear probe for kitchen or participant identity on sink tokens versus local tokens.

The second stage is the intervention. For a training clip of class $c$ with $n_c$ examples, we mask its sink tokens with probability

$$ p_c = p_{\max}\left(1 - \left(\frac{n_c}{n_{\max}}\right)^{\gamma}\right), $$

where $n_{\max}$ is the largest class count, $p_{\max}$ caps the masking rate, and $\gamma$ controls how quickly masking rises toward the tail. When a clip is masked, we add a consistency term $\lambda\,\mathrm{KL}(f(x)\,\|\,f_{\setminus s}(x))$ with stop-gradient on the full-token prediction. Here $f$ uses all tokens, $f_{\setminus s}$ uses all tokens except the sinks, and $\lambda$ weights the term. This keeps the masked and unmasked pathways in agreement rather than splitting them.

Inference is unchanged. For the video-text model the class count $n_c$ comes from verb and noun frequencies in the narrations, which is natural in EPIC-style data. The contrastive loss is otherwise left as InfoNCE, so the effect of the token intervention is isolated.

Baselines

The naive reference point is a plain fine-tune of the same backbone with cross-entropy, or with InfoNCE for retrieval. This tells us how much of the tail gap is due to the shortcut at all. A second simple reference is class-balanced sampling paired with balanced softmax. The file warns that loss reweighting alone can underperform: Class-Balanced Loss gave 51.1% mAP against a 51.6% baseline on VideoLT (VideoLT). Because it is cheap, every method, including ours, will also be stacked on top of it.

The strongest existing competitor is LMR from Use Your Head. It reports 39.7% average class accuracy on EPIC-KITCHENS-100 against 21.2% for cross-entropy, and 35.7% few-shot accuracy where standard models score 0.0%. The backbone behind those numbers is not given in the file, so we will reimplement LMR on our exact ViT. We will use the same data, epochs, augmentation, and a matched hyperparameter search of about twenty trials per method.

On VideoLT, the strongest reported figure is MEID at 61.9% overall mAP with ResNet-50 (MEID). Again we will rerun it on our ViT features rather than compare across backbones. For retrieval, LaViLa reports only relative gains in the file, a 5.9% absolute mAP gain on multi-instance retrieval. No reported absolute result was found on Lacuna for LaViLa's MIR mAP, so we will establish our own LaViLa baseline.

A video version of CMO, which pastes tail-clip foreground volumes onto head-clip backgrounds, is the most direct rival hypothesis and is discussed below.

Three ablations isolate the key ingredient. The first masks an equal number of random tokens at the same rarity-dependent rate. The second masks sinks uniformly regardless of class. The third deletes sinks entirely at both training and inference. Beating LMR means higher few-shot and tail accuracy at matched head accuracy, measured over three seeds. The decisive bar is that the random-token ablation must fail to match our method. Otherwise, the gains are just generic regularization.

Why not just…?

Why not just paste rare actions into varied backgrounds, CMO-style, extended to video? This is the real threat, because it attacks the same context shortcut. Its weaknesses are documented. Pasting creates boundary artifacts and semantic nonsense when spatial context is ignored, a limitation the file notes for CMO. In video, the pasted foreground must also be tracked coherently through time. We will build a video CMO with tube-shaped pastes and run it under the same budget. If it matches our tail gains on unseen environments, the honest conclusion is that the token-level account adds explanation but no practical value. We will report that.

Why not just remove or regularize sink tokens for everyone, or add register tokens? Sinks carry useful scene semantics: routing them separately gave a 15-point gain on MME code reasoning over LLaVA-7B (To Sink or Not to Sink). Head classes may legitimately use scene context. The uniform-masking and full-deletion ablations test whether rarity-dependence matters. Our prediction is that both cost head accuracy, which is the "robbing Peter to pay Paul" trade-off familiar from long-tail work.

Why not just unfreeze the backbone and rebalance? MOVE shows that frozen features suffer a domain gap on rare categories (MOVE), so full fine-tuning is a fair alternative. However, fine-tuning on few tail clips is precisely when memorizing scene context is easiest. We run every method both with a frozen backbone plus adapter and with full fine-tuning. The diagnostic in stage one tells us whether unfreezing reduces or increases sink reliance.

Why not just reweight the loss or use balanced softmax? These adjust how much the tail counts, not what evidence the model uses. VideoLT shows that reweighting alone can fall below an unweighted baseline. We include it under every method rather than as a substitute.

Proposed Experiments

The main benchmark is EPIC-KITCHENS-100 action recognition. We report top-1 accuracy, average class accuracy, and few-shot accuracy, following LMR's protocol of at most 20 training examples per class. The decisive split is clips from environments or participants under the environmental shift the benchmark documents. We will also run SSv2-LT, where context is deliberately uninformative. There we predict little sink reliance and little gain, which serves as a built-in negative control. VideoLT is used for head, medium, and tail mAP at scale.

For video-text, we evaluate EPIC-KITCHENS-100 multi-instance retrieval split by query verb and noun frequency. We train the LaViLa-style encoder with human narrations only and with synthetic narrations added. This tests whether synthetic text, which may hallucinate common entities, deepens tail reliance on scene tokens.

The decisive checks are three. First, sink reliance must rise toward the tail in stage one. Second, the frequency-dependent intervention must beat both random-token masking and uniform sink masking on tail accuracy in unseen environments, without losing head accuracy. Third, it must match or exceed our reimplemented LMR and video CMO under equal compute, with near-zero added training cost relative to SV-Mix's 52% throughput loss. If the first check fails, the work stops and we publish the negative diagnosis. If the third fails, the contribution is reduced to the diagnostic.