Introduction: The Challenge of Video Redundancy
Vision-Language Models (VLMs) like GPT-4o and LLaVA have demonstrated remarkable capabilities in multimodal tasks. However, video understanding presents a unique challenge: extreme data redundancy. A typical video contains thousands of frames, yet the "smoking gun" evidence required to answer a specific reasoning question often resides in just a few key moments. Processing every frame is computationally prohibitive ("token-hungry"), while naive downsampling often misses critical information.
Existing keyframe selection methods typically rely on visual relevance, identifying frames that contain objects or actions mentioned in the query. However, "looking relevant" does not equate to being "causally informative." For instance, if asked why an event occurred, a frame showing the event's aftermath is visually relevant but lacks the causal context needed for an accurate explanation.
Beyond Visual Relevance: The ReaSon Framework
To address this, we present ReaSon (Reinforced Causal Search with Information Bottleneck). ReaSon shifts the focus from visual similarity to causal necessity—identifying the specific moments without which the correct reasoning chain would fail.
As shown in the figure above, a standard visual search might select frames of a woman drinking water because they match the query's keywords. However, to understand why she is drinking, the model must identify the causal context: she is preparing for a medical examination. ReaSon identifies these "causally necessary" frames (marked with red stars) to provide a grounded, accurate answer.
Methodology: Causal Information Bottleneck
ReaSon treats keyframe selection as a reinforcement learning (RL) problem optimized via a Causal Information Bottleneck (CIB). The framework comprises two primary modules:
- Predictive Sufficiency Module: This module generates a candidate pool of frames. It utilizes a detector (YOLO-World) to locate visual elements related to the question and an LSTM-based policy network to select a compact subset of frames sufficient for generating a correct answer.
- Causal Necessity Module: This module acts as a filter for causal significance. It employs counterfactual interventions, asking: "If these frames were replaced, how significantly would the VLM's confidence in the answer drop?" Frames that cause a high divergence in the output distribution when removed are deemed causally necessary.
The system is trained using a composite reward function that balances accuracy ($R_{ans}$), cycle consistency ($R_{cycle}$), and causal necessity ($R_{cf}$).
Experimental Results
ReaSon was evaluated across three major benchmarks, consistently achieving state-of-the-art (SOTA) performance:
- NExT-QA: Achieved 81.4% accuracy using LLaVA-Video-7B, outperforming previous specialized methods.
- EgoSchema: Reached 72.2% accuracy with GPT-4o, demonstrating superior performance in long-form egocentric video reasoning.
- Video-MME: ReaSon demonstrated exceptional efficiency; using only 32 frames, it outperformed models utilizing hundreds of frames, proving that the quality of frame selection is more critical than quantity.
Discussion and Future Directions
The success of ReaSon highlights that correlation is a poor proxy for reasoning in video understanding. By explicitly training models to distinguish between frames that merely correlate with an answer and those that cause the model to reach that answer, ReaSon significantly reduces hallucinations.
Current limitations include a reliance on internal model knowledge and difficulties in tracking entities over very long durations. Future iterations will explore integrating external knowledge retrieval and dedicated long-term memory modules to further enhance causal reasoning.
Summary updated