TL;DR
This paper introduces Video-Text Interleaved Chain-of-Thought (ViTCoT), a novel video reasoning paradigm that embeds key visual frames directly within the step-by-step textual reasoning process of Multimodal Large Language Models (MLLMs). By mimicking the human tendency to re-examine visual evidence while thinking, ViTCoT significantly improves video understanding and activates deeper neural engagement across various model architectures.
What Problem Does This Paper Address?
Chain-of-Thought (CoT) prompting has revolutionized how Large Language Models tackle complex reasoning tasks by breaking them down step-by-step. When applied to video understanding, however, current MLLMs suffer from a major cognitive disconnect: they ingest the video at the beginning but perform their step-by-step reasoning entirely in the text domain. This ignores the visual modality during the actual problem-solving phase.
In contrast, humans naturally look back at specific parts of a video to verify details while reasoning through a complex question. The authors address this gap by creating a paradigm where visual and textual tokens are interleaved throughout the reasoning chain, allowing the model to "look back" at key moments while it computes its answer.
Methods
The authors propose ViTCoT, a flexible, two-stage video-text interleaved reasoning framework that can be integrated into existing mainstream CoT methods (like Vanilla CoT, Description CoT, and Plan-and-Solve):
- Initial Text Reasoning: The MLLM is provided with the original video, a question, and multiple-choice options. It is instructed to generate a preliminary, rough textual reasoning framework without directly outputting the final answer.
- Video-Text Interleaved Reasoning: Task-relevant "key-videos" (sequences of keyframes) are dynamically embedded or interleaved directly into this initial text reasoning chain. The model then processes this combined multimodal context to generate the final, refined reasoning steps and the correct answer.
Datasets
To evaluate this new paradigm, the authors developed the Video-Text Interleaved Benchmark (ViTIB), built on top of the existing VideoEspresso dataset.
- Statistics: ViTIB contains 1,382 videos and 5,051 keyframes across 14 distinct everyday reasoning categories (e.g., Causal Analysis, Cooking Process, Traffic Analysis). On average, each video features a curated "key-video" subset of 3.7 frames.
- Construction Pipeline: The dataset was built by filtering out incomplete samples, using Gemini-2.0-Flash to automatically recognize and extract the most task-relevant keyframes, and finally applying a rigorous three-person human verification process to ensure every selected frame directly supports the correct answer.
Key Insights and Ideas
- Visual-Textual Semantic Overlap: Using CLIP to encode both the keyframes and the textual reasoning answers, the authors showed via UMAP dimensionality reduction that keyframe features almost entirely encompass the feature space of the textual reasoning content. This proves that the selected keyframes effectively capture the cognitive "essence" of the reasoning process.
- Deeper Model Engagement: Mechanistic analysis revealed that ViTCoT triggers significantly more neuron activations and attention head engagement across layers in MLLMs (like Qwen2.5-VL-3B) compared to traditional text-only CoT. Interleaving visual tokens literally forces the model to utilize more of its internal capacity.
- Redundancy Reduction: Interestingly, evaluating models using only the extracted key-videos often outperformed using the full, original videos. This implies that the key-video extraction successfully filters out irrelevant visual noise, making reasoning more efficient.
Key Results
The authors evaluated ViTCoT across several prominent open-source and closed-source MLLMs, including Qwen2.5-VL (3B/7B), VideoLLaMA3-7B, Intern2.5-VL-8B, and Gemini-2.0-Flash.
- Consistent Performance Boosts: ViTCoT methods consistently outperformed their text-only vanilla CoT counterparts. On Qwen2.5-VL-7B-Instruct, ViTCoT achieved an average accuracy improvement of 5.4% over vanilla CoT.
- Superiority Over Simple Concatenation: To prove that the improvement wasn't just from giving the model more data, they tested a baseline where both the original video and key-video were fed to the model at the very beginning. ViTCoT (interleaved) still beat this baseline by an average margin of 2.8% on Gemini-2.0-Flash, proving that how the video is presented (interleaved vs. concatenated) matters immensely.
Limitations and Future Work
While the main experiments rely on an "Oracle" key-video (the gold-standard keyframes from the benchmark), real-world deployment requires automated keyframe extraction. The authors conducted a preliminary test using unsupervised CLIP retrieval to select rough keyframes based on the initial text reasoning, which still yielded a 1.7% improvement. However, developing more robust, training-free, or end-to-end models that can dynamically and accurately self-extract and interleave these keyframes during inference remains an important direction for future work.
Summary updated