Video language models (systems that watch a clip and answer questions about it in text) name the objects in a scene accurately but often invent the actions. They see snow and say "skiing," or see a ball and say "throwing" when the person is catching. We propose a small, interpretable side model that settles, for each action verb the language model is about to produce, whether the verb is supported by actual motion in the video or only by the scenery that usually accompanies it. The side model is a joint non-negative matrix factorization, a decomposition that rewrites a data matrix as a sum of parts using only positive building blocks. It is fit on video features and caption verbs. During decoding it gently discounts verbs whose support comes from scene context alone. This takes one extra matrix projection per video, not extra forward passes of the language model.
The opportunity comes from a specific gap. Current fixes for temporal hallucination either contrast the model against corrupted copies of the video, which roughly doubles or triples inference cost, or retrain the architecture on hundreds of thousands of examples. Neither tells you why a verb was chosen. The vocabulary-mismatch literature names the underlying problem: continuous motion has to be mapped onto a discrete verb vocabulary, and when that mapping is weak the decoder falls back on stereotyped word associations. An additive factorization is the natural tool for making that mapping explicit, because it can say how much of a verb's evidence comes from each source without positive and negative contributions cancelling out.
Core Claim
A block-structured joint factorization can learn two kinds of components from captioned video. Scene components are tied to appearance features and verbs. Motion components are tied to order-sensitive frame-change features and verbs. The two kinds compete to explain each verb's occurrences. Verbs that merely co-occur with settings get absorbed by scene components, and verbs that track real movement get absorbed by motion components. At test time, the share of a verb's predicted support that comes from motion components is a per-video, per-verb grounding score. Using that score to reweight verb tokens during decoding should reduce action and direction hallucination at close to baseline latency, and do so at least as well as contrastive decoding.
The claim that would be new is the attribution itself. No cited method separates scene-driven from motion-driven verb evidence in a decomposition that a human can inspect and that leaves the language model untouched.
Relation to Prior Work
The method comes from MCA-NMF, which concatenates non-negative histograms of speech, gesture, and vision into one matrix and factorizes it so that an agent learns word–object–motion associations without supervision. We reuse that cross-modal dictionary idea, with three changes. We replace shallow histograms with features from a modern video model's vision encoder. We impose a block structure that keeps scene and motion components apart. And we connect the result to autoregressive decoding, which MCA-NMF never addresses. Two further supports for the design choice come from NMF's topic-modeling record. It works well on short texts, as shown in Federated NMF for short texts, and captions are short. It also preserves long-tail topics, as shown in A Topic Coverage Approach, and rare verbs are exactly where hallucination concentrates. To choose the number of components we borrow the silhouette-based "Binary Bleed" search of SMART-SLIC. That work uses non-negative tensor factorization to ground text-only retrieval and never touches video.
The motivating evidence comes from three diagnoses. EventHallusion shows that video models get objects right and fabricate events. MASH-VLM traces action–scene hallucination to attention that mixes spatial and temporal tokens. VidHalluc documents action, sequence, and scene-transition errors across 5,002 videos. MASH-VLM fixes the mixing with masked attention and roughly 430k fine-tuning examples. We pursue the same separation without retraining, as a linear-algebraic decomposition outside the network.
Several existing methods constrain the design. SEASON builds a "temporally homogenized" negative by averaging features across frames and contrasts against it, which raises latency. SmartSight warns that contrastive decoding can penalize legitimate reasoning, so our intervention touches only verb tokens. Temporal-Aware Activation Engineering uses only a binary temporal-variant/invariant label, which motivates our graded per-verb score. DINO-HEAL, from VidHalluc, reweights visual tokens by foreground saliency and admittedly fails on scene transitions. Our motion features include background change, so transitions are not invisible to them.
Research Program
Features. For each training clip we pool the video model's own vision-encoder tokens per frame. We take the clip-mean as the scene feature $s$. The motion feature $m$ is built from consecutive-frame differences, with each signed difference split into its positive and negative parts so that everything stays non-negative. Reversing the clip swaps these parts, so motion direction is encoded rather than averaged away. The verb vector $t$ counts lemmatized verbs in the clip's captions. Training clips come from a captioned video corpus kept strictly disjoint from all benchmark videos.
Block factorization. Stack clips as rows and fit
$$[S \mid M \mid T] \approx [W_s \mid W_m]\begin{bmatrix} H^{s}_{S} & 0 & H^{s}_{T} \\ 0 & H^{m}_{M} & H^{m}_{T} \end{bmatrix}.$$
Here $S, M, T$ are the scene, motion, and verb blocks. $W_s, W_m \ge 0$ are clip loadings on scene and motion components. The zero blocks forbid scene components from explaining motion features and motion components from explaining scene features. Both kinds may explain verbs, so a verb ends up in whichever component type predicts it better. All factors are non-negative, and blocks are weighted so that each modality contributes comparable reconstruction error.
Grounding score. For a test video we fix $H$ and solve a non-negative least-squares fit of $(w_s, w_m)$ to its scene and motion features only, without verbs. The motion share for verb $v$ is
$$r_v = \frac{(w_m H^{m}_{T})_v}{(w_s H^{s}_{T})_v + (w_m H^{m}_{T})_v + \epsilon},$$
where $\epsilon$ is a small constant. During decoding, at any step where a candidate token begins a lexicon verb, we add $\alpha \log(r_v + \epsilon)$ to its logit. The strength $\alpha$ is tuned on a held-out split. Non-verb tokens, and therefore reasoning text, are left untouched.
Analysis. We inspect the learned components directly. One check is whether verbs such as "skiing" or "surfing" split their mass between scene and motion components in plausible proportions. Another is whether direction-sensitive pairs such as catching/throwing and opening/closing land on motion components whose positive and negative difference parts are mirrored.
Baselines
The naive reference point is the unmodified base model, LLaVA-OV-7B (the model on which SEASON reports latency), decoded greedily on each benchmark. Every other method must beat it, and its latency is the cost floor.
The strongest existing training-free competitor is SEASON's self-diagnostic contrastive decoding. It targets the same temporal failures and also constructs a scene-preserving, order-erasing negative. On cost, SEASON reports per-sample decoding latency rising from 0.80s to 1.21s on LLaVA-OV-7B. We found no reported accuracy for SEASON on VidHalluc, EventHallusion, VidHal, or MESH on Lacuna, so we will rerun it ourselves with its released settings. Beating it means matching or exceeding its accuracy on action and direction subsets at latency close to the 0.80s baseline. We also run Temporal Contrastive Decoding from EventHallusion, DINO-HEAL from VidHalluc, and SmartSight's sampling with early termination from SmartSight. No accuracy figures for these are reported on Lacuna either. Fairness requires the same base model, frames per video, and prompts, plus the same held-out tuning budget for each method's single strength hyperparameter, with tuning on a split disjoint from test.
Three ablations isolate the key ingredient. First, a motion-only probe: a multi-label logistic regression from $m$ to verbs, trained on the same clips and captions, whose predicted probability replaces $r_v$. Second, an unblocked NMF with no zero constraints, where scene and motion share components. Third, a "frame-average negative" variant that scores verbs by how much worse they are predicted from $s$ alone than from $(s,m)$, which is SEASON's homogenization idea moved into feature space. If the block NMF does not beat the probe and the unblocked NMF, the additive attribution is not what is helping.
Why not just…?
Why not just use contrastive decoding such as TCD or SEASON? These methods work and are the bar to clear. Their weaknesses are cost and coverage. SEASON adds extra forward passes, measured at 0.80s→1.21s. SmartSight reports that contrastive penalties misfire on multi-step reasoning, and frame-dropping in TCD is a blunt way to erase order. Our method adds no language-model passes and touches only verbs. Whether that narrower intervention gives up accuracy is an empirical question, and the head-to-head on action and direction subsets will answer it.
Why not just train a classifier from motion features to verbs? This is the most serious threat, since a motion-only probe also avoids scene shortcuts. Our argument is that a probe outputs a verb probability without saying whether the corpus association behind that verb was really scene-driven. Its signed weights can cancel, so it cannot cleanly report how much of "skiing" is snow. The block NMF forces scene and motion to compete for each verb and returns an additive share. If the probe matches the block NMF on the benchmarks, we will report that the useful ingredient is motion-only verb evidence, not the factorization, and present the NMF's contribution as interpretability alone.
Why not just fine-tune with counterfactual preference data, as in CounterVid? CounterVid generates videos that differ only in action or order and trains with preference optimization. MASH-VLM likewise retrains. Both are likely stronger in absolute terms and require video synthesis or large fine-tuning sets. We position our method as a training-free, inspectable alternative and will report the gap honestly where public checkpoints allow a comparison, not claim parity.
Why not just prompt the model to describe the motion before answering? The reasoning traces themselves hallucinate. Yan et al. and Zenz et al. find fabricated claims in reasoning even when answers are right, and longer responses drift away from visual tokens (Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context). We include a describe-motion-first prompt as a cheap baseline.
Proposed Experiments
We evaluate on the action, temporal-sequence, and scene-transition subsets of VidHalluc, on EventHallusion, on VidHal, and on MESH, reporting each benchmark's native accuracy and per-sample latency. To check that object grounding and general ability are not harmed, we also report POPE-style object questions run on video frames, plus MM-Vet on still images.
The method should be judged on three decisive checks. First, on the action and direction subsets the block NMF must beat both the motion-only probe and the unblocked NMF. If it does not, the attribution claim fails. Second, on a reversed-video control, where each test clip is played backwards, the grounding score for direction-sensitive verbs should flip, and accuracy on reversed clips should hold up better than under the base model. This tests whether the split positive and negative difference features carry order. Third, the improvement should be largest on clips whose scene strongly predicts a wrong verb, such as snow without skiing. We identify these clips beforehand as those where scene-component support dominates. A uniform gain across all clips would suggest that generic verb damping, not attribution, is doing the work. We will also report the scene-transition subset separately, since it is where DINO-HEAL fails and where background-change features should matter most.
References
- MCA-NMF: Multimodal Concept Acquisition with Non-Negative Matrix Factorization
- Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information
- A Topic Coverage Approach to Evaluation of Topic Models
- Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization
- EventHallusion: Diagnosing Event Hallucinations in Video LLMs
- MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations
- VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding
- SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding
- SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse
- Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering
- CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models
- Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models
- Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- VidHal: Benchmarking Temporal Hallucinations in Vision LLMs
- MESH - Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
- Evaluating Object Hallucination in Large Vision-Language Models
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities