BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind

Yuanyuan Mao, Xin Lin, Qin Ni, Liang He

Abstract

Current Video Question Answering (VideoQA) models excel at physical causal reasoning but struggle with social intelligence. We introduce BDIQA, a dataset designed to evaluate an agent's "Theory of Mind" by testing its understanding of Belief, Desire, and Intention (BDI). Using synthetic 3D environments, BDIQA reveals a significant performance gap between human reasoning and state-of-the-art AI, particularly in scenarios involving false beliefs.

Introduction

Human social interaction is predicated on the ability to attribute mental states to others—a cognitive capacity known as Theory of Mind (ToM). While existing VideoQA benchmarks focus on objective physical interactions (e.g., "What object was moved?"), they often overlook the underlying cognitive drivers of behavior. For instance, if a person searches an empty container for a snack, a human observer understands this action through the lens of a false belief. Current AI models, however, frequently fail to distinguish between what is physically happening and what a character thinks is happening.

The BDIQA dataset addresses this gap by providing a structured framework to evaluate whether AI can infer the internal states of agents in complex social scenes.

Methodology: The BDI Framework

BDIQA is built upon a two-level hierarchy inspired by developmental psychology:

  • Level 0 (Simple Reasoning): Scenarios involving satisfied desires, straightforward intentions, and true beliefs where the character's internal map matches the physical world.
  • Level 1 (Complex Reasoning): Scenarios involving unsatisfied desires and false beliefs. A classic example is an object being moved while a character is out of the room; the character subsequently acts based on outdated information.
A character interacting with a refrigerator in the VirtualHome environment, illustrating a typical BDIQA scenario.

To ensure precise control over ground-truth mental states, the authors utilized VirtualHome, a simulation platform for household activities. This allows for the generation of multi-agent interactions where intentions and knowledge states are explicitly programmed and tracked.

The BDIQA Dataset

The dataset comprises 3,527 videos and nearly 20,000 QA pairs. It features two primary characters, Alice and Job, performing various household tasks across 50 distinct environment layouts.

A character in a dining room setting, demonstrating the 3D animated nature of the BDIQA dataset.

Questions are categorized into five types: Belief, Desire, Intention, "Where" (perception-based), and "Yes/No" (verification). Unlike 2D grid-based datasets, BDIQA's 3D animations provide a more realistic bridge toward human-AI interaction in physical spaces.

Key Insights and Results

Evaluations of various models—ranging from Graph Neural Networks to large-scale pre-trained transformers like ClipBERT and Frozen—yielded several critical findings:

  1. The Perception-Reasoning Gap: Models performed well on "Where" questions but struggled with "Belief" questions. This indicates that while models can track object coordinates, they fail to model the character's internal representation of those objects.
  2. The "False Belief" Challenge: Performance plummeted when transitioning from Level 0 to Level 1 tasks. Understanding that an agent can hold a mistaken view of reality remains a significant hurdle for neural networks.
  3. Limitations of General Pre-training: Massive models trained on general web data (e.g., Frozen) showed poor zero-shot performance, often falling below 1% accuracy. This suggests that Theory of Mind reasoning is not an emergent property of standard video-language pre-training and requires specific logical grounding.
  4. Human vs. Machine: While the best-performing model (Comem) reached approximately 79.84% accuracy, it still lagged behind the human baseline of 84.23%, particularly in complex reasoning tasks.

Conclusion

BDIQA highlights a fundamental deficiency in current AI: the lack of a robust cognitive model for human behavior. While perception is the necessary basis for reasoning, it is not sufficient for social intelligence. Future work must bridge the "sim-to-real" gap and explore architectures that incorporate psychological theories of mind to create agents capable of true collaboration with humans.

Summary updated

Abstract

As a foundational component of cognitive intelligence, theory of mind (ToM) can make AI more closely resemble human thought processes, thereby enhancing their interaction and collaboration with human. In particular, it can significantly improve a model's comprehension of videos in complex scenes. However, current video question answer (VideoQA) datasets focus on studying causal reasoning within events, few of them genuinely incorporating human ToM. Consequently, there is a lack of development in ToM reasoning tasks within the area of VideoQA. This paper presents BDIQA, the first benchmark to explore the cognitive reasoning capabilities of VideoQA models in the context of ToM. BDIQA is inspired by the cognitive development of children's ToM and addresses the current deficiencies in machine ToM within datasets and tasks. Specifically, it offers tasks at two difficulty levels, assessing Belief, Desire and Intention (BDI) reasoning in both simple and complex scenarios. We conduct evaluations on several mainstream methods of VideoQA and diagnose their capabilities with zero-shot, few-shot and supervised learning. We find that the performance of pre-trained models on cognitive reasoning tasks remains unsatisfactory. To counter this challenge, we undertake thorough analysis and experimentation, ultimately presenting two guidelines to enhance cognitive reasoning derived from ablation analysis.
BibTeX
@inproceedings{mao2024bdiqa,
  author = {Yuanyuan Mao and Xin Lin and Qin Ni and Liang He},
  title = {{BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27814},
}
Figure