Abstract
IntentQA introduces a new challenge for AI: moving beyond recognizing what is happening in a video to reasoning about why it is happening. The researchers contribute a large-scale dataset focused on human intentions and propose CaVIR, a model that integrates situational, contrastive, and commonsense contexts to decode the "dark matter" of human behavior.
Introduction: The "Dark Matter" of Human Behavior
Most Video Question Answering (VideoQA) systems excel at "factoid" tasks—identifying objects, colors, or explicit actions (e.g., "What is the person holding?"). However, they struggle with "inference" tasks that require understanding latent variables like mental states and goals.
The core challenge is that human intent is highly context-sensitive. For example, a person pointing at a cup could mean "Give me water" in a restaurant, "Clean this" near a sink of dirty dishes, or "Look at this art" in a gallery. To solve this, models need to look beyond the immediate action and incorporate situational surroundings, contrastive examples of similar actions with different goals, and general world knowledge (commonsense).
Methodology: The CaVIR Model
The authors propose the Context-aware Video Intent Reasoning (CaVIR) model, which tackles the problem through three specialized modules:
- Situational Context (Video Query Language - VQL): This module uses a graph-based approach to align specific video regions with the text of the question and candidate answers. It ensures the model "looks" at the parts of the video most relevant to the query.
- Contrastive Context (Contrastive Learning): To help the model distinguish between similar actions with different intents, the authors use a triplet loss mechanism. During training, the model compares an "anchor" video with a "positive" sample (similar intent) and a "negative" sample (similar action but different intent).
- Commonsense Context (LLM Integration): Recognizing that intent often relies on social norms, the model queries a Large Language Model (GPT) to provide a "prior" probability for the answers based purely on the text. This linguistic prior is then fused with the visual features to reach a final decision.
The IntentQA Dataset
The researchers introduced the IntentQA dataset, derived from the existing NExT-QA benchmark. It consists of 4,303 videos and 16,297 question-answer pairs. The dataset focuses on four types of reasoning: Causal Why (CW), Causal How (CH), Temporal Previous (TP), and Temporal Next (TN).
Annotation involved a rigorous multi-stage process using dependency parsing to extract key actions and Amazon Mechanical Turk (AMT) for human verification, ensuring that every question specifically targets human intent.
Key Insights and Results
- Intent as "Dark Matter": The paper treats intent as the invisible force driving visible actions. The uncertainty doesn't come from the physical movement, but from the environment and social context.
- The Power of Contrast: One of the most effective ways to learn intent is to see the same action lead to different outcomes. By explicitly training on these "minimal pairs" via contrastive learning, the model becomes much more robust.
- State-of-the-Art Performance: The full CaVIR model achieved 57.64% accuracy on the IntentQA test set, a significant improvement over the base model (51.36%) and previous SOTA models like VGT (51.27%).
- The Human Gap: Despite these gains, there is still a massive "intelligence gap." Human performance on the same dataset is 78.49%, indicating that intent reasoning remains a formidable frontier for AI.
Limitations and Future Outlook
Temporal reasoning (TP & TN) remains the hardest category. While causal questions (Why/How) have more explicit visual cues, predicting what happened before or after a clip requires a deeper level of mental simulation. Future work could involve more sophisticated ways of modeling social interactions and subtle non-verbal cues like gaze or micro-expressions that signal human intent.
Summary updated