Abstract
This survey provides a comprehensive overview of the evolution of Visual Question Answering (VQA), tracing its development from early pattern-matching models to the current state-of-the-art Multimodal Large Language Models (MLLMs). We categorize VQA methodologies into three main stages: conventional deep learning, vision-language pre-training, and MLLMs. By examining the shift from simple perception to complex cognitive inference, this paper highlights how MLLMs leverage external knowledge and zero-shot reasoning to bridge the gap between visual understanding and natural language reasoning.
Introduction
Visual Question Answering (VQA) represents a cornerstone challenge in artificial intelligence, requiring the seamless integration of computer vision and natural language processing. Historically, VQA was approached as a classification task, where models identified objects to select answers from a fixed set. However, true human-level VQA necessitates deeper cognition—the ability to infer context, apply common sense, and reason over multiple steps. This survey maps the transition from traditional deep learning architectures to the transformative era of MLLMs, highlighting the shift from simple image-text alignment to sophisticated cognitive inference.
Taxonomy of VQA Components
The VQA landscape is structured around two primary pillars: Natural Language Understanding (NLU) of Image and Text, and Natural Language Inference (NLI). Understanding involves the extraction of visual and textual features followed by sophisticated fusion mechanisms. Inference, on the other hand, focuses on knowledge acquisition from structured or unstructured sources and the subsequent reasoning process, which can range from conventional logic to multi-hop reasoning.
Methodological Evolution
The paper categorizes the evolution of VQA into three distinct stages:
- Conventional Deep Learning: Early models utilized Convolutional Neural Networks (CNNs) for image processing and Recurrent Neural Networks (RNNs/LSTMs) for text. These features were fused through vector operations or basic attention mechanisms.
- Vision-Language Pre-training (VLP): Models such as CLIP, ViLBERT, and LXMERT introduced a middle ground, pre-training on massive datasets to align visual and textual features before task-specific fine-tuning.
- Multimodal Large Language Models (MLLMs): This current paradigm includes LLM-aided systems (where the LLM manages specialized tools), Image-to-Text approaches (generating captions for text-only LLMs), and General MLLMs (end-to-end models like LLaVA or Flamingo).
Benchmarking and Datasets
The survey reviews a wide array of benchmarks categorized by their intent. Standard benchmarks like VQAv1/v2 and GQA focus on foundational and compositional reasoning. Knowledge-based datasets like OK-VQA and FVQA require external information not present in the image. Newer MLLM-specific benchmarks, such as MME and MathVista, test high-level logic and mathematical reasoning while monitoring for "hallucination" tendencies.
Key Insights and Results
A major trend in MLLMs is the "Frozen" advantage, where a massive language model remains frozen while a small alignment module (e.g., BLIP-2’s Q-Former) is trained. This preserves the LLM's reasoning capabilities while granting it visual perception. Furthermore, reasoning has evolved from "one-hop" fact identification to "multi-hop" evidence connection.
Performance metrics show significant gains:
- VQAv2: Modern MLLMs like mPLUG-Huge achieve accuracies up to 83.62%, far exceeding older attention-based models (~70%).
- OK-VQA: PaLI-X leads with 66.1%, demonstrating improved world-knowledge integration.
- Zero-Shot: MLLMs like BLIP-2 achieve ~65% on VQAv2 without specific training, a feat previously impossible for earlier generations.
Future Challenges
Despite progress, challenges remain in robustness (overcoming language priors), explainability (the "black box" nature of MLLMs), and hallucination (stating non-existent objects). Improving fine-grained perception for small details and complex spatial relationships remains a critical area for future research.
Summary updated