Abstract
Current Video Question Answering (VideoQA) models excel at physical causal reasoning but struggle with social intelligence. We introduce BDIQA, a dataset designed to evaluate an agent's "Theory of Mind" by testing its understanding of Belief, Desire, and Intention (BDI). Using synthetic 3D environments, BDIQA reveals a significant performance gap between human reasoning and state-of-the-art AI, particularly in scenarios involving false beliefs.
Introduction
Human social interaction is predicated on the ability to attribute mental states to others—a cognitive capacity known as Theory of Mind (ToM). While existing VideoQA benchmarks focus on objective physical interactions (e.g., "What object was moved?"), they often overlook the underlying cognitive drivers of behavior. For instance, if a person searches an empty container for a snack, a human observer understands this action through the lens of a false belief. Current AI models, however, frequently fail to distinguish between what is physically happening and what a character thinks is happening.
The BDIQA dataset addresses this gap by providing a structured framework to evaluate whether AI can infer the internal states of agents in complex social scenes.
Methodology: The BDI Framework
BDIQA is built upon a two-level hierarchy inspired by developmental psychology:
- Level 0 (Simple Reasoning): Scenarios involving satisfied desires, straightforward intentions, and true beliefs where the character's internal map matches the physical world.
- Level 1 (Complex Reasoning): Scenarios involving unsatisfied desires and false beliefs. A classic example is an object being moved while a character is out of the room; the character subsequently acts based on outdated information.
To ensure precise control over ground-truth mental states, the authors utilized VirtualHome, a simulation platform for household activities. This allows for the generation of multi-agent interactions where intentions and knowledge states are explicitly programmed and tracked.
The BDIQA Dataset
The dataset comprises 3,527 videos and nearly 20,000 QA pairs. It features two primary characters, Alice and Job, performing various household tasks across 50 distinct environment layouts.
Questions are categorized into five types: Belief, Desire, Intention, "Where" (perception-based), and "Yes/No" (verification). Unlike 2D grid-based datasets, BDIQA's 3D animations provide a more realistic bridge toward human-AI interaction in physical spaces.
Key Insights and Results
Evaluations of various models—ranging from Graph Neural Networks to large-scale pre-trained transformers like ClipBERT and Frozen—yielded several critical findings:
- The Perception-Reasoning Gap: Models performed well on "Where" questions but struggled with "Belief" questions. This indicates that while models can track object coordinates, they fail to model the character's internal representation of those objects.
- The "False Belief" Challenge: Performance plummeted when transitioning from Level 0 to Level 1 tasks. Understanding that an agent can hold a mistaken view of reality remains a significant hurdle for neural networks.
- Limitations of General Pre-training: Massive models trained on general web data (e.g., Frozen) showed poor zero-shot performance, often falling below 1% accuracy. This suggests that Theory of Mind reasoning is not an emergent property of standard video-language pre-training and requires specific logical grounding.
- Human vs. Machine: While the best-performing model (Comem) reached approximately 79.84% accuracy, it still lagged behind the human baseline of 84.23%, particularly in complex reasoning tasks.
Conclusion
BDIQA highlights a fundamental deficiency in current AI: the lack of a robust cognitive model for human behavior. While perception is the necessary basis for reasoning, it is not sufficient for social intelligence. Future work must bridge the "sim-to-real" gap and explore architectures that incorporate psychological theories of mind to create agents capable of true collaboration with humans.
Summary updated