Abstract
As a foundational component of cognitive intelligence, theory of mind (ToM) can make AI more closely resemble human thought processes, thereby enhancing their interaction and collaboration with human. In particular, it can significantly improve a model's comprehension of videos in complex scenes. However, current video question answer (VideoQA) datasets focus on studying causal reasoning within events few of them genuinely incorporating human ToM. Consequently, there is a lack of development in ToM reasoning tasks within the area of VideoQA. This paper presents BDIQA, the first benchmark to explore the cognitive reasoning capabilities of VideoQA models in the context of ToM. BDIQA is inspired by the cognitive development of children's ToM and addresses the current deficiencies in machine ToM within datasets and tasks. Specifically, it offers tasks at two difficulty levels, assessing Belief, Desire and Intention (BDI) reasoning in both simple and complex scenarios. We conduct evaluations on several mainstream methods of VideoQA and diagnose their capabilities with zero shot, few shot and supervised learning. We find that the performance of pre-trained models on cognitive reasoning tasks remains unsatisfactory. To counter this challenge, we undertake thorough analysis and experimentation, ultimately presenting two guidelines to enhance cognitive reasoning derived from ablation analysis.
Introduction
Human intelligence relies heavily on Theory of Mind (ToM): the cognitive ability to attribute mental states to others and recognize that those states might differ from our own. Most current Video Question Answering (VideoQA) benchmarks focus on physical causal reasoning or simple action description. However, if a character walks toward an empty fridge because they don't know someone else took the food, a human understands they have a false belief. Current AI models often fail this "Sally-Anne" style reasoning, describing actions without understanding underlying motivations or incorrect assumptions. This paper introduces BDIQA, a benchmark designed to bridge the gap between visual perception and high-level cognitive reasoning.
The BDIQA Dataset
The researchers introduced BDIQA, a new VideoQA dataset designed to evaluate an AI's ability to reason about a character's Beliefs, Desires, and Intentions (BDI).
The dataset was generated using VirtualHome, a simulation platform for household activities, and consists of 3,527 videos and 19,932 QA pairs. It is structured into two difficulty levels:
- Level 0: Simple reasoning with satisfied desires and true beliefs.
- Level 1: Complex scenarios involving unsatisfied desires, multi-step plans, and false beliefs.
Methodology
The study evaluates machine ToM through three pillars:
- Belief: What the character thinks is true (including "false beliefs" when the environment changes out of sight).
- Desire: The character's ultimate goal (e.g., "having a meal").
- Intention: Specific plans executed to fulfill a desire (e.g., "switching on the stove").
The authors tested various architectures, including standard memory-based models (HME, Comem), Graph Neural Networks (HGA, Dual), and large-scale pre-trained transformers (Frozen, JustAsk, ClipBERT). They also proposed enhancements such as integrating a memory module to facilitate multi-step reasoning and upgrading visual backbones (ResNet/ResNeXt) to improve perception.
Key Results and Insights
The evaluation revealed several critical findings:
- Perception as a Foundation: Improving the visual backbone significantly boosted reasoning performance, confirming that accurate object tracking is a prerequisite for mental state attribution.
- Zero-Shot Failure: Large pre-trained models performed poorly (0-1% accuracy) in zero-shot settings on BDI tasks. This suggests ToM is not an emergent property of general video training but requires specific cognitive modeling.
- The Memory Requirement: Adding a memory module was essential for handling temporal dependencies, allowing models to remember past states and link them to current actions.
- Performance Gap: The Comem model achieved the best baseline performance (79.84% accuracy). By adding a memory module and better visual features, the JustAsk model reached 81.01%, approaching the human baseline of ~84%. However, a significant gap remains in complex cognitive reasoning (Acc_bdi), where humans still outperform the best models.
Conclusion
BDIQA highlights the current limitations of AI in understanding human mental states. While models are improving at perception and simple temporal reasoning, the "false belief" problem and intrinsic motivation inference remain significant challenges. Future work will focus on expanding the dataset to more complex real-world scenarios and developing architectures specifically tailored for high-level cognitive reasoning.
Summary updated