IntentQA: Context-aware Video Intent Reasoning

Jiapeng Li, Ping Wei, Wenjuan Han, Lifeng Fan

Abstract

IntentQA introduces a new challenge for AI: moving beyond recognizing what is happening in a video to reasoning about why it is happening. The researchers contribute a large-scale dataset focused on human intentions and propose CaVIR, a model that integrates situational, contrastive, and commonsense contexts to decode the "dark matter" of human behavior.

Introduction: The "Dark Matter" of Human Behavior

Most Video Question Answering (VideoQA) systems excel at "factoid" tasks—identifying objects, colors, or explicit actions (e.g., "What is the person holding?"). However, they struggle with "inference" tasks that require understanding latent variables like mental states and goals.

The core challenge is that human intent is highly context-sensitive. For example, a person pointing at a cup could mean "Give me water" in a restaurant, "Clean this" near a sink of dirty dishes, or "Look at this art" in a gallery. To solve this, models need to look beyond the immediate action and incorporate situational surroundings, contrastive examples of similar actions with different goals, and general world knowledge (commonsense).

Figure 1: The same action (pointing to a cup) can have vastly different intents depending on the situational and commonsense context.

Methodology: The CaVIR Model

The authors propose the Context-aware Video Intent Reasoning (CaVIR) model, which tackles the problem through three specialized modules:

  1. Situational Context (Video Query Language - VQL): This module uses a graph-based approach to align specific video regions with the text of the question and candidate answers. It ensures the model "looks" at the parts of the video most relevant to the query.
  2. Contrastive Context (Contrastive Learning): To help the model distinguish between similar actions with different intents, the authors use a triplet loss mechanism. During training, the model compares an "anchor" video with a "positive" sample (similar intent) and a "negative" sample (similar action but different intent).
  3. Commonsense Context (LLM Integration): Recognizing that intent often relies on social norms, the model queries a Large Language Model (GPT) to provide a "prior" probability for the answers based purely on the text. This linguistic prior is then fused with the visual features to reach a final decision.

The IntentQA Dataset

The researchers introduced the IntentQA dataset, derived from the existing NExT-QA benchmark. It consists of 4,303 videos and 16,297 question-answer pairs. The dataset focuses on four types of reasoning: Causal Why (CW), Causal How (CH), Temporal Previous (TP), and Temporal Next (TN).

Figure 4: Examples from the IntentQA dataset illustrating the four reasoning categories: Causal Why (CW), Causal How (CH), Temporal Next (TN), and Temporal Previous (TP).

Annotation involved a rigorous multi-stage process using dependency parsing to extract key actions and Amazon Mechanical Turk (AMT) for human verification, ensuring that every question specifically targets human intent.

Key Insights and Results

  • Intent as "Dark Matter": The paper treats intent as the invisible force driving visible actions. The uncertainty doesn't come from the physical movement, but from the environment and social context.
  • The Power of Contrast: One of the most effective ways to learn intent is to see the same action lead to different outcomes. By explicitly training on these "minimal pairs" via contrastive learning, the model becomes much more robust.
  • State-of-the-Art Performance: The full CaVIR model achieved 57.64% accuracy on the IntentQA test set, a significant improvement over the base model (51.36%) and previous SOTA models like VGT (51.27%).
  • The Human Gap: Despite these gains, there is still a massive "intelligence gap." Human performance on the same dataset is 78.49%, indicating that intent reasoning remains a formidable frontier for AI.

Limitations and Future Outlook

Temporal reasoning (TP & TN) remains the hardest category. While causal questions (Why/How) have more explicit visual cues, predicting what happened before or after a clip requires a deeper level of mental simulation. Future work could involve more sophisticated ways of modeling social interactions and subtle non-verbal cues like gaze or micro-expressions that signal human intent.

Summary updated

Abstract

In this paper, we propose a novel task IntentQA, a special VideoQA task focusing on video intent reasoning, which has become increasingly important for AI with its advantages in equipping AI agents with the capability of reasoning beyond mere recognition in daily tasks. We also contribute a large-scale VideoQA dataset for this task. We propose a Context-aware Video Intent Reasoning model (CaVIR) consisting of i) Video Query Language (VQL) for better cross-modal representation of the situational context, ii) Contrastive Learning module for utilizing the contrastive context, and iii) Commonsense Reasoning module for incorporating the commonsense context. Comprehensive experiments on this challenging task demonstrate the effectiveness of each model component, the superiority of our full model over other baselines, and the generalizability of our model to a new VideoQA task. The dataset and codes are open-sourced at: https://github.com/JoseponLee/IntentQA.git.
BibTeX
@inproceedings{li2023intentqa,
  author = {Jiapeng Li and Ping Wei and Wenjuan Han and Lifeng Fan},
  title = {{IntentQA: Context-aware Video Intent Reasoning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01099},
}
Figure