When a general video-language model is fine-tuned to caption videos from one narrow domain, such as cooking or film camera work, the domain itself hands the model an easy cheat. In cooking footage a knife and a board nearly always mean "slice," so a captioner can learn to name the action from one still frame without ever looking at the motion. We propose to detect this cheat during specialization, using only the model's own predictions and no extra labels. The test is simple: run an early, briefly fine-tuned version of the model on a single frame and on the full clip, and flag the caption words it can already predict from the still. A second training run then keeps the ordinary captioning loss and adds one requirement for the flagged action words: the model must find them more likely given the real, ordered clip than given a frozen, shuffled, or reversed version of it.
This matters because two bodies of evidence pull against each other. Specializing compact models usually wins on domain benchmarks. Yet the same domain fine-tuning can also teach the model shortcuts, and video models already tend to read narratives off keyframes. The method takes the "train once to find failures, then retrain with them upweighted" recipe from shortcut-learning research. It moves that recipe from image classification to open-ended caption generation, applies it at the level of individual words, and runs it inside parameter-efficient domain fine-tuning. The two-sentence version is this. Domain fine-tuning makes video captioners guess actions from static scenery. We find the guessed words with a single-frame probe on an early checkpoint, then retrain so those words must be supported by the video's actual temporal order.
Core Claim
The claim has two parts. The first is that standard parameter-efficient domain fine-tuning of a video captioner increases its reliance on static frames. Concretely, the share of verb and direction tokens that the model can predict from a single frame should rise over training, even as caption metrics like CIDEr (a consensus score against reference captions) improve. The second is that this shortcut is best located at the token level, not the sample level, and best repaired with a temporal counterfactual margin, not by dropping or upweighting whole videos.
For a training video $V$ with reference caption tokens $y_1,\dots,y_T$, define a static-sufficiency score from an early-stopped domain model $p_0$:
$$s_t = \log p_0(y_t \mid v^{*}, y_{<t}) - \log p_0(y_t \mid V, y_{<t}).$$
Here $v^{*}$ is one randomly chosen frame tiled to the clip length. A score $s_t$ near or above zero means the still frame already explains token $y_t$. The flagged set $\mathcal{S}$ contains tokens with $s_t$ above a threshold that are also motion-bearing words, meaning verbs, direction words, and temporal connectives. A part-of-speech tagger on the reference caption decides which words count as motion-bearing. This restriction is needed because objects are legitimately recognizable from a still, and penalizing them would be wrong.
The second-stage model $p_\theta$ is retrained from the base checkpoint with this objective:
$$\mathcal{L} = -\sum_t (1+\lambda\,\mathbb{1}[t\in\mathcal{S}])\log p_\theta(y_t\mid V,y_{<t}) + \beta\sum_{t\in\mathcal{S}}\max\big(0,\ m - \log p_\theta(y_t\mid V,y_{<t}) + \log p_\theta(y_t\mid \tilde V,y_{<t})\big).$$
In this objective, $\tilde V$ is a counterfactual clip: a frozen frame, shuffled frames, or the reversed clip. The quantity $m$ is a likelihood margin, and $\lambda,\beta$ are weights. The first term is the error-set upweighting of Just Train Twice, rewritten for tokens. The second term is a temporal, generative analogue of the natural-direct-effect regularizer in See or Guess.
Relation to Prior Work
Three lines of evidence motivate the gap. Unhackable Temporal Reward names "temporal hacking," in which a model infers a clip's caption from one keyframe. It measures this with Temporal Perplexity and fixes it by building new training data from external trackers (Grounding DINO, SAM2). Its setting is general-domain QA. Our work differs in three ways: it uses the hackability signal per token as a training weight, it needs no tracker, and it studies whether domain specialization specifically creates the hack. Why Can't I Open My Drawer? shows that action classifiers learn objects quickly and verbs slowly, so accuracy on unseen verb-object pairs falls toward zero during training. Its fix (object mixing, a forward/reverse playback regularizer, a margin on frequent pairs) is built for classification. We borrow the reversal idea and apply it to a caption decoder. STSBench finds that driving-specialized models trail a generalist: DriveMM scores 39.51% and InternVL 2.5 8B scores 46.07%. This is the clearest sign in the file that specialization can hurt temporal reasoning, though that paper proposes no training fix.
Some prior work points the other way and must be faced directly. Domain-Specific Semantics Guided Video Captioning trains a decoder per domain and reports gains on MSVD and MSR-VTT. ShotBench shows that compact cinematography-specialized models beat GPT-4o, with ShotVL-7B at 70.1% against 59.3%. AgriPerceiver shows that a small trained bridge on frozen backbones beats larger generalists. Our view is that these results and STSBench do not contradict each other. Average metrics can rise while temporal grounding erodes, and ShotBench itself notes that camera movement remains the weakest dimension. Our first experiment tests exactly this.
Other work supplies parts of the method. Just Train Twice supplies the two-stage structure and the finding that static upweighting beats dynamic robust optimization. See or Guess supplies the counterfactual-effect loss. That loss was built for images with bounding-box masks, and we replace the mask with temporal corruption. Learning to Decode Against Compositional Hallucination uses similar perturbations (frame shuffling, blurring), but only at inference time through contrastive decoding. VideoPASTA uses error-injected captions as negatives for preference tuning, with a 32B teacher filtering the data. Our negatives are corrupted videos, not corrupted text, and no teacher model is involved.
Research Program
The first stage is a diagnostic study. We start from an open Video-LLM of roughly 7B parameters. Candidates are VideoLLaMA3-7B and Qwen3-VL-8B, both of which TriCD evaluates. We specialize the model with low-rank adapters on two domains. The first is procedural cooking on YouCook2, a domain with strong utensil-to-action co-occurrence. The second is film camera movement, using clips from the film domain covered by ShotBench, paired with captions written from its camera-movement labels. At evenly spaced checkpoints we record four quantities. Two are caption quality measures, CIDEr and METEOR. The third is the static-sufficiency rate: the fraction of motion tokens with $s_t > 0$ on held-out data. The fourth is a reversal flip rate: on reversed clips with an action that has an antonym ("open/close," "pan left/pan right," "add/remove"), how often the generated verb changes. Our prediction is that CIDEr rises while static sufficiency rises and the flip rate falls. If this does not happen, the motivation fails and we report that.
The second stage is the two-stage method described above. Stage 1 is a short adapter run, early-stopped by training loss, not by any group labels, because the need for validation labels is a known weakness of JTT. Stage 2 retrains from the base model. The tagger, threshold, $\lambda$, $\beta$, and $m$ are tuned on a held-out split, scored by the flip rate and CIDEr. The flip rate is computed automatically from antonym lists, not from human group labels.
The third stage is a transfer check. After specialization, we test whether the model still handles general temporal benchmarks. This addresses the specialization-induced collapse that the file flags as an open problem.
Baselines
The naive reference point has two parts. One is the base Video-LLM prompted zero-shot to caption each domain. The other is plain adapter fine-tuning with the standard caption loss, which is the usual specialization recipe. The second is the most important comparison, because our central claim is that this recipe causes the shortcut.
The strongest specialization-side competitor is the ShotVL recipe from ShotBench: supervised fine-tuning followed by group relative policy optimization, a reinforcement learning method that scores groups of sampled outputs against each other. We run it on the camera-movement domain with the same base model and adapter rank. Its reward is camera-movement correctness parsed from the caption, and it gets the same GPU-hour budget as our two stages combined. ShotBench reports 70.1% average accuracy for ShotVL-7B over eight dimensions. No camera-movement-only score is reported on Lacuna beyond the remark that over half of tested models fall below 40%. We therefore re-measure the camera-movement subset ourselves.
The strongest shortcut-side competitor is the temporal-hacking approach of Unhackable Temporal Reward. Video-UTR-7B reports 59.67% on TempCompass and 58.78% on MVBench. We run a tracker-free version that drops or downweights the training clips with the lowest Temporal Perplexity, at the same compute budget. This isolates the sample-level versus token-level question.
We also run three further comparisons. The first is sequence-level JTT, which upweights whole captions with high stage-1 loss. The second is VideoPASTA-style preference tuning with text-corrupted negatives. The third is inference-time contrastive decoding in the style of TriCD applied to the plain fine-tuned model, which reports gains of 9.61% and 12.03% on OmniVCHall for the two base models. All methods share the same data, the same base checkpoint, the same frame budget, and an equal hyperparameter search of twenty trials.
For dense captioning context, Streaming Dense Video Captioning reports CIDEr 41.2 on ActivityNet Captions and 32.9 on YouCook2. That is a dense-captioning setup, so we cite it as context, not as a matched baseline.
Three ablations isolate the key ingredients. The first applies the margin to all motion tokens, with no stage-1 flagging. The second flags tokens but drops the part-of-speech filter. The third compares the counterfactual types against each other: frozen frame, shuffle, and reverse. Beating the baselines means two things at once. Reversal flip rate and held-out-composition verb accuracy must improve significantly over plain fine-tuning and over Temporal Perplexity filtering. CIDEr must fall by no more than one point.
Why not just…?
Why not just use a bigger generalist with more frames? Evidence in the file argues against scale alone. Unhackable Temporal Reward reports an anti-scaling effect, where more capacity memorizes static cues. TempCompass finds direction and speed near chance (25–33%) for most video models. GPT-4o trails ShotVL-7B on ShotBench. We still include a larger generalist in the diagnostic study, since a generalist with a low flip-rate gap would undercut the motivation.
Why not just filter out hackable training clips? This is the most serious alternative. In domains with little data, removing clips also removes the vocabulary they teach. And a single clip often contains both legitimately static words (objects) and shortcut words (verbs), so sample-level filtering is too coarse. The head-to-head against Temporal Perplexity filtering at equal compute settles this. If filtering matches us on both flip rate and CIDEr, the token-level contribution is not supported, and we will say so.
Why not just run JTT on caption loss? Caption loss is high for many reasons, chiefly rare vocabulary and annotator variance, not shortcut failure. So the sequence-level error set may mostly upweight rare words. We measure how much the JTT error set overlaps with our static-sufficiency set, and compare the two on flip rate.
Why not just fix it at decoding time with contrastive decoding? TriCD leaves the weights unchanged and adds extra forward passes for every output. If it recovers the same flip-rate gains on the plain fine-tuned model, training-time repair is unnecessary. We report both its accuracy and its inference cost.
Decisive Checks
Two results decide the proposal. The first is the diagnostic curve: under plain adapter fine-tuning, static sufficiency must rise while CIDEr rises. Without this, there is no specialization-induced shortcut to fix. The second is a held-out-composition test on YouCook2, built from verb-object pairs that are rare or absent in training. This is the captioning analogue of a worst group. The full method must raise verb accuracy there over plain fine-tuning, Temporal Perplexity filtering, and sequence-level JTT, while holding CIDEr and METEOR on the standard test split. Secondary checks are TempCompass and MVBench scores after specialization, to confirm that general temporal skill is preserved, and the ShotBench camera-movement subset, where the method should exceed the ShotVL-style recipe at matched compute. If the ablation with no flagging matches the full method, the error-set idea is unnecessary. The contribution would then reduce to a temporal margin loss for captioners, and we would report it that way.
References
- Just Train Twice: Improving Group Robustness without Training Group Information
- See or Guess: Counterfactually Regularized Image Captioning
- Unhackable Temporal Reward for Scalable Video MLLMs
- Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
- Domain-Specific Semantics Guided Approach to Video Captioning
- ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models
- AgriPerceiver: A Parameter-Efficient Vision-Language Model for Structured Macroscopic Crop Phenotyping
- Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models
- VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment
- Streaming Dense Video Captioning
- TempCompass: Do Video LLMs Really Understand Videos?