TL;DR
VIVID moves video reasoning beyond static keyframe analysis by explicitly modeling inter-frame changes and tracking entity histories. By injecting "Visual Deltas" into existing tokens and maintaining "Entity State Chains," VIVID achieves state-of-the-art performance on motion-heavy benchmarks using only 0.22% trainable parameters—even when calibrated on as few as 50 examples.
What Problem Does This Paper Address?
Current Video Multimodal Large Language Models (MLLMs) often treat videos as a collection of independent static snapshots. This approach suffers from two major bottlenecks:
- Spatial Bottleneck: Standard vision encoders use whole-frame pooling, which dilutes small moving objects (like a soccer ball) into the static background. If the background doesn't change, the model might miss the motion entirely.
- Temporal Bottleneck: Models typically use uniform sampling (e.g., one frame every second). This often misses brief, critical "action peaks"—like a hand grabbing a tool—that occur between the sampled frames.
As shown in Figure 1, these bottlenecks lead to models confusing the direction of motion or miscounting repeated actions because they literally "blink" and miss the most important parts of the video.
Methods
The authors propose VIVID (Visual Inter-frame Variations and Individual Dynamics), an architecture designed to mimic human vision by combining sparse fixations with high sensitivity to motion.
1. Autonomous Delta-Driven Keyframe Planning (ADKP)
Instead of uniform sampling, ADKP analyzes a 3 fps candidate pool to identify frames with the highest local patch changes. It selects $K_f$ "transition anchors" that align with actual action peaks rather than arbitrary timestamps.
2. Dual-Road Change Memory (DRCM)
This is the core of VIVID, processing information through two specialized pathways:
- Visual Delta Pathway: It computes "Patch Diffs" (the difference between consecutive frames). These differences are compressed and injected directly into the keyframe embeddings as Dynamic Tokens. Crucially, this adds zero extra tokens to the LLM's context window, preserving efficiency. It also crops a camera-stabilized Active Motion ROI to focus on small moving targets.
- Entity State Pathway: A vision-language "writer" generates chronological text descriptions of entity transitions (e.g., "Arm reaches for part #1"). These chains provide a symbolic memory that helps the model reason about long-term event order and repetition.
3. Transition Conditioned Adaptation
VIVID uses a massive 27B parameter frozen backbone (Qwen). To adapt it for video, the authors only train the small Visual Delta Module and apply LoRA to the final 16 layers of the LLM, totaling just 0.22% of the original parameters.
Datasets
VIVID was evaluated against eleven baselines across three major benchmarks:
- MotionBench: 4,017 questions focused on fine-grained motion (direction, speed, repetition).
- Video-MME: 2,700 questions across short, medium, and long-form videos.
- TempCompass: 1,366 questions designed to test temporal reasoning (action, order, attribute change).
Training was conducted at two scales: VIVID-Small (calibrated on just 50 examples) and VIVID-Large (trained on 3,600 clips from TVBench and Charades-STA).
Key Insights and Ideas
- Deltas are more informative than frames: By focusing on what changed between frames rather than just the frames themselves, the model can detect subtle motions that global pooling normally erases.
- Zero-Overhead Dynamics: The "Dynamic Tokens" approach is a clever engineering feat. By folding motion residuals into existing anchor tokens, the model gains temporal awareness without increasing the computational cost of the LLM's self-attention layers.
- Few-Shot Power: The fact that VIVID-Small beats all eleven baselines (including those trained on much larger datasets) after seeing only 50 examples suggests that the inductive bias of modeling deltas is extremely powerful for video understanding.
- Human-Centric Design: The combination of an Active Motion ROI (like a foveal fixation) and Entity State Chains (like working memory) closely mirrors how humans process complex visual scenes.
Key Results
VIVID-Large achieved the highest overall accuracy across all three benchmarks.
- MotionBench: VIVID-Large reached 62.78% overall accuracy, significantly outperforming the static uniform sampling baseline (58.85%) and specialized models like VideoStreaming and ProtoKV.
- Efficiency: VIVID-Small, despite its tiny training set, outperformed all eleven baselines in overall accuracy (61.71%).
- Temporal Precision: VIVID showed massive gains in "Repetition Count" and "Action Order," categories where traditional models typically struggle.
Limitations and Future Work
- Candidate Pool Limits: ADKP currently selects anchors from a 3 fps pool. While this captures most human actions, extremely high-speed motion (e.g., a bullet or a fast-moving insect) might still be missed if it falls between the 0.33s intervals.
- Entity State Bottleneck: The Entity State Pathway relies on a vision-language writer. If the initial description of an entity is incorrect, that error can propagate through the state chain.
- Future Directions: The authors suggest exploring even more efficient ways to scale to hour-long videos by further amortizing the cost of building Entity State Chains across multiple queries.
Summary updated