Abstract
Visual Question Answering (VQA) is a challenging task that requires cross-modal understanding and reasoning of visual image and natural language question. To inspect the association of VQA models to human cognition, we designed a survey to record human thinking process and analyzed VQA models by comparing the outputs and attention maps with those of humans. We found that although the VQA models resemble human cognition in architecture and performs similarly with human on the recognition-level, they still struggle with cognitive inferences. The analysis of human thinking procedure serves to direct future research and introduce more cognitive capacity into modeling features and architectures.
Introduction
Visual Question Answering (VQA) represents a significant milestone in multimodal artificial intelligence, requiring a machine to synthesize visual data with natural language queries. While modern models have achieved high accuracy on standard benchmarks, there is a growing concern regarding their cognitive depth. This study investigates whether these models are truly "reasoning" or merely exploiting statistical biases. By comparing a baseline model and a state-of-the-art (SOTA) model against human cognitive processes, we aim to identify the gap between computational performance and genuine understanding.
Methodology
Our comparative study involved three distinct agents: a baseline ResNet-LSTM model with basic attention, the SOTA OSCAR model, and a group of 21 human participants from NYU.
Model Architectures
The baseline model utilizes a traditional combination of convolutional layers for vision and recurrent units for text. In contrast, the OSCAR model (Object-Semantics Aligned Pre-training) leverages a transformer-based architecture to more effectively align word tokens with specific image regions and object tags.
Human Survey Design
To capture human cognition, we designed a survey where participants did not just provide answers but also identified the image regions they prioritized and explained their reasoning. This allowed for a direct comparison between model "attention maps" and human "thought paths."
Experimental Setup and Datasets
Models were trained on standard VQA 1.0 and v2.0 benchmarks. For the comparative analysis, we curated a custom dataset consisting of five representative images from MS COCO, paired with 20 questions categorized by cognitive demand: Object-oriented (recognition), Scene-oriented (relationships), Counting, and Knowledge-oriented (inference).
Key Insights and Results
The "Common Sense" Gap
Our findings highlight a significant disparity in "common sense" reasoning. Humans naturally utilize domain knowledge to infer context—such as identifying a rainy scene from umbrellas even when raindrops are invisible. Models frequently fail these knowledge-oriented tasks because they lack a comprehensive world model.
Recognition vs. Reasoning
While models excel at recognition-level tasks (e.g., identifying a dog), they struggle with cognitive inference. For instance, when presented with a baby holding a phone, models often predicted "brushing teeth" or "eating ice cream," misinterpreting the hand-to-face gesture because they failed to understand the specific object (the phone) in context.
Attention and Alignment
The SOTA OSCAR model showed significantly higher alignment with human answers than the baseline, particularly in object recognition. However, human attention remains more "intentional," focusing on semantically rich areas like faces, whereas model attention is often scattered or biased toward the largest objects in a frame.
Conclusion and Future Work
While VQA models are becoming proficient at pattern matching, they still lack the reasoning depth of human intelligence. Future research should focus on:
- Narrative Generation: Enabling models to explain the logic behind their answers.
- Cognitive Scaling: Moving from simple object detection to reasoning that incorporates broader knowledge bases.
- Human-Inspired Architectures: Designing attention mechanisms that better mimic the global-to-local shifts seen in human vision.
Summary updated