VIVID: Beyond Static Multi-Image Reasoning with Localized Visual Differences and Textual Entity Dynamics in Video MLLMs

Chengqian Ma, Yuanze Zeng, Hongyu Wang, Mingfeng Lin, YiCheng Xiao, Yu Chen, Pengyu Li, WenHao Wang, Yuexiao Ma, Xiawu Zheng, Fei ChaoarXiv 2026 · arXiv

TL;DR

VIVID moves video reasoning beyond static keyframe analysis by explicitly modeling inter-frame changes and tracking entity histories. By injecting "Visual Deltas" into existing tokens and maintaining "Entity State Chains," VIVID achieves state-of-the-art performance on motion-heavy benchmarks using only 0.22% trainable parameters—even when calibrated on as few as 50 examples.

What Problem Does This Paper Address?

Current Video Multimodal Large Language Models (MLLMs) often treat videos as a collection of independent static snapshots. This approach suffers from two major bottlenecks:

  1. Spatial Bottleneck: Standard vision encoders use whole-frame pooling, which dilutes small moving objects (like a soccer ball) into the static background. If the background doesn't change, the model might miss the motion entirely.
  2. Temporal Bottleneck: Models typically use uniform sampling (e.g., one frame every second). This often misses brief, critical "action peaks"—like a hand grabbing a tool—that occur between the sampled frames.

As shown in Figure 1, these bottlenecks lead to models confusing the direction of motion or miscounting repeated actions because they literally "blink" and miss the most important parts of the video.

Figure 1. Spatial and temporal bottlenecks of static multi image reasoning, and how VIVID resolves both (shown at Kf = 4 for clarity; Kf = 8 in benchmarks): (a) Spatial Bottleneck: whole frame pooling dilutes small moving targets into static backgrounds, whereas VIVID crops the Active Motion Region of Interest (ROI) and folds surrounding patch differences into Dynamic Tokens; (b) Temporal Bottleneck: equal interval sampling steps over brief action peaks, whereas ADKP captures both reaches and precomputed Entity State Chains count repeated actions accurately.

Methods

The authors propose VIVID (Visual Inter-frame Variations and Individual Dynamics), an architecture designed to mimic human vision by combining sparse fixations with high sensitivity to motion.

1. Autonomous Delta-Driven Keyframe Planning (ADKP)

Instead of uniform sampling, ADKP analyzes a 3 fps candidate pool to identify frames with the highest local patch changes. It selects $K_f$ "transition anchors" that align with actual action peaks rather than arbitrary timestamps.

2. Dual-Road Change Memory (DRCM)

This is the core of VIVID, processing information through two specialized pathways:

  • Visual Delta Pathway: It computes "Patch Diffs" (the difference between consecutive frames). These differences are compressed and injected directly into the keyframe embeddings as Dynamic Tokens. Crucially, this adds zero extra tokens to the LLM's context window, preserving efficiency. It also crops a camera-stabilized Active Motion ROI to focus on small moving targets.
  • Entity State Pathway: A vision-language "writer" generates chronological text descriptions of entity transitions (e.g., "Arm reaches for part #1"). These chains provide a symbolic memory that helps the model reason about long-term event order and repetition.

3. Transition Conditioned Adaptation

VIVID uses a massive 27B parameter frozen backbone (Qwen). To adapt it for video, the authors only train the small Visual Delta Module and apply LoRA to the final 16 layers of the LLM, totaling just 0.22% of the original parameters.

Figure 2. Overview of the VIVID architecture (ViT: Vision Transformer; MLP: multilayer perceptron; FFN: feedforward network; ROI: region of interest; LoRA: Low Rank Adaptation): (a) across 3 fps candidates (Sec. 3), the frozen ViT computes Patch Diffs ∆Et so ADKP (Eqs. (6) to (8)) selects Kf = 8 transition anchors across intervals B1 . . . B6 and crops a 320 × 180 Active Motion ROI; (b) surrounding Patch Diffs enter Dual Road Change Memory (DRCM), where the Visual Delta Module θ∆enriches static anchors Eˆtk into Dynamic Tokens ˜Eˆtk (Eqs. (3) and (4)) alongside precomputed Entity State Chains (Eq. (5)); (c) the Transition Conditioned Input feeds the shared reader, with the first 48 LLM layers frozen and the final 16 layers adapted via LoRA.

Datasets

VIVID was evaluated against eleven baselines across three major benchmarks:

  • MotionBench: 4,017 questions focused on fine-grained motion (direction, speed, repetition).
  • Video-MME: 2,700 questions across short, medium, and long-form videos.
  • TempCompass: 1,366 questions designed to test temporal reasoning (action, order, attribute change).

Training was conducted at two scales: VIVID-Small (calibrated on just 50 examples) and VIVID-Large (trained on 3,600 clips from TVBench and Charades-STA).

Key Insights and Ideas

  • Deltas are more informative than frames: By focusing on what changed between frames rather than just the frames themselves, the model can detect subtle motions that global pooling normally erases.
  • Zero-Overhead Dynamics: The "Dynamic Tokens" approach is a clever engineering feat. By folding motion residuals into existing anchor tokens, the model gains temporal awareness without increasing the computational cost of the LLM's self-attention layers.
  • Few-Shot Power: The fact that VIVID-Small beats all eleven baselines (including those trained on much larger datasets) after seeing only 50 examples suggests that the inductive bias of modeling deltas is extremely powerful for video understanding.
  • Human-Centric Design: The combination of an Active Motion ROI (like a foveal fixation) and Entity State Chains (like working memory) closely mirrors how humans process complex visual scenes.

Key Results

VIVID-Large achieved the highest overall accuracy across all three benchmarks.

  • MotionBench: VIVID-Large reached 62.78% overall accuracy, significantly outperforming the static uniform sampling baseline (58.85%) and specialized models like VideoStreaming and ProtoKV.
  • Efficiency: VIVID-Small, despite its tiny training set, outperformed all eleven baselines in overall accuracy (61.71%).
  • Temporal Precision: VIVID showed massive gains in "Repetition Count" and "Action Order," categories where traditional models typically struggle.
Table 1. Per category accuracy (%) across the six official motion categories of MotionBench on the shared Qwen3.8 27B backbone under the unified dynamic 3 fps candidate pool (M(T) ∈[8, 96] →Kf = 8 reader frames; bold indicates best and underline second best).

Limitations and Future Work

  • Candidate Pool Limits: ADKP currently selects anchors from a 3 fps pool. While this captures most human actions, extremely high-speed motion (e.g., a bullet or a fast-moving insect) might still be missed if it falls between the 0.33s intervals.
  • Entity State Bottleneck: The Entity State Pathway relies on a vision-language writer. If the initial description of an entity is incorrect, that error can propagate through the state chain.
  • Future Directions: The authors suggest exploring even more efficient ways to scale to hour-long videos by further amortizing the cost of building Entity State Chains across multiple queries.

Summary updated

Abstract

Video question answering under tight frame budgets requires preserving localized motion and event order, yet current video multimodal large language models (MLLMs) reduce comprehension to static multi-image reasoning by discarding unselected frames and encoding keyframes independently: (1) Spatially, whole-frame pooling dilutes small moving targets into static backgrounds; (2) Temporally, dropping intermediate frames or averaging tokens erases inter-frame motion, confusing direction, speed, and order. We propose VIVID (Visual Inter-frame Variations and Individual Dynamics), which moves beyond static snapshots like human vision: keyframes capture core states while inter-frame Patch Diffs preserve neighboring changes within $K_f$ keyframes. Autonomous Delta-Driven Keyframe Planning (ADKP) selects anchors from a $3\text{ fps}$ pool and feeds surrounding patch differences into Dual-Road Change Memory (DRCM), forming Dynamic Tokens at zero extra anchor tokens alongside a camera-stabilized Active Motion Region of Interest (ROI) and chronological Entity State Chains. Adding $0.22\%$ trainable parameters to a frozen $27\text{B}$ backbone, VIVID-Small ($50$ examples, $\sim 1.5$ GPU hours) beats all eleven baselines in overall accuracy across $8,083$ questions and trails only VIVID-Large ($3,600$ examples), which leads on MotionBench, Video-MME, and TempCompass.
BibTeX
@misc{ma2026vivid,
  author = {Chengqian Ma and Yuanze Zeng and Hongyu Wang and Mingfeng Lin and YiCheng Xiao and Yu Chen and Pengyu Li and WenHao Wang and Yuexiao Ma and Xiawu Zheng and Fei Chao},
  title = {{VIVID: Beyond Static Multi-Image Reasoning with Localized Visual Differences and Textual Entity Dynamics in Video MLLMs}},
  year = {2026},
  eprint = {2611.23153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23153},
}
Figure