TL;DR
This survey paper provides a comprehensive, up-to-date overview of the Visual Question Answering (VQA) landscape, charting its evolution from conventional deep learning architectures to modern Multimodal Large Language Models (MLLMs). By systematically classifying the field into perceptual (understanding) and cognitive (inference) dimensions, the authors highlight how VQA has transitioned from simple feature matching to sophisticated multi-hop reasoning and instruction tuning.
What Problem Does This Paper Address?
Visual Question Answering (VQA) sits at the challenging intersection of Computer Vision (CV) and Natural Language Processing (NLP). The core task is simple to state—predict an accurate natural language answer given an image (or video) and a related question—but incredibly difficult to execute.
Early VQA systems focused primarily on basic perception: extracting visual features from an image, textual features from a question, and fusing them to predict an answer. However, real-world VQA requires more than just "seeing"; it demands cognitive reasoning, context awareness, and often the integration of external common-sense knowledge. With the recent explosion of Multimodal Large Language Models (MLLMs), the paradigm of VQA has fundamentally shifted. This paper addresses the need for a unified, timely taxonomy that bridges traditional deep learning fusion strategies with the latest zero-shot, few-shot, and generative reasoning capabilities of MLLMs.
Methods
The authors organize the VQA framework into a clear taxonomy divided into two primary phases: Natural Language Understanding of Image and Text and Natural Language Inference.
1. Multimodal Understanding (Perception)
The paper outlines the progression of feature extraction and fusion:
- Conventional Deep Learning: Early approaches relied on CNNs (like VGG or ResNet) for images and RNNs/LSTMs for text, combining them via simple vector operations or bilinear pooling. This evolved into attention mechanisms (co-attention) and Graph Neural Networks (GNNs) to model spatial and semantic relationships.
- Vision-Language Pre-training (VLP): Models like ViLBERT and LXMERT introduced dual-stream architectures, while UNITER and ViLT pioneered single-stream architectures to learn joint representations on massive unlabeled datasets.
- MLLM Paradigms: The survey identifies three main ways modern MLLMs comprehend visual data:
- LLM-Aided Visual Understanding: The LLM acts as a central controller, breaking down the question and dispatching tasks to external vision tools (e.g., Visual ChatGPT).
- Image-to-Text Generation: Transforming the image into captions or OCR text before feeding it directly into a text-only LLM (e.g., PICa).
- General Multimodal LLMs: End-to-end networks where visual features are directly mapped into the LLM's textual space using alignment modules like Q-Former or Perceiver Resampler (e.g., BLIP-2, LLaVA).
2. Natural Language Inference (Cognition)
For reasoning and answer generation, the methods span:
- Knowledge-Based Reasoning: Extracting internal or external knowledge (from sources like DBpedia or ConceptNet) and performing one-hop or multi-hop reasoning using memory networks or knowledge graphs.
- MLLM Reasoning: Leveraging advanced prompting techniques such as multimodal instruction tuning, in-context learning (ICL), and Multimodal Chain-of-Thought (M-CoT) to generate step-by-step rationales before arriving at an answer.
Datasets
The survey categorizes VQA benchmarks based on their reliance on external knowledge:
- Datasets without External Knowledge: Classic benchmarks like DAQUAR, VQAv1, VQAv2, CLEVR (synthetic geometry), GQA, and specialized tasks like DocVQA and VizWiz.
- Datasets with External Knowledge Base: Benchmarks explicitly requiring outside information, including KB-VQA, FVQA, OK-VQA, and KVQA.
- MLLM Benchmarks: The paper highlights a new breed of comprehensive evaluation suites like MME, SEED-BENCH, MathVista, and MMMU, which assess MLLMs across highly specialized domains (e.g., mathematics, physics, and logic).
Key Insights and Ideas
- From Perception to Cognition: VQA has evolved from a "what you see is what you get" task into a complex cognitive challenge. The most successful modern architectures focus heavily on how knowledge is retrieved and structured for multi-step deduction.
- The Power of Frozen LLMs: A key architectural trend in MLLMs is keeping the heavy language backbone frozen while training lightweight alignment modules (like adapters or Q-Formers). This drastically lowers computational costs while inheriting the LLM's powerful text-reasoning capabilities.
- Shortcuts vs. Grounding: Traditional deep learning models frequently suffered from "language bias" (e.g., answering "yellow" to any question starting with "What color is the banana?"). While MLLMs mitigate this through better cross-modal grounding, they introduce new risks, such as hallucinating objects not present in the scene.
Key Results
The paper provides a comprehensive performance meta-analysis across four benchmark datasets (VQAv2, OK-VQA, GQA, and VizWiz):
- VQAv2: MLLMs achieve state-of-the-art results, with mPLUG-Huge leading at 83.62% accuracy, compared to early deep learning baselines like MUTAN at 67.4%.
- OK-VQA (Knowledge-Intensive): PaLI-X achieves the highest accuracy at 66.1%, significantly outperforming specialized graph-based reasoning models like Mucko (29.2%), proving that massive scale and unified pre-training can implicitly capture vast amounts of external knowledge.
- VizWiz: Models like PaLI achieve up to 73.3% accuracy on real-world, low-quality images taken by visually impaired users.
Limitations and Future Work
Despite massive progress, the authors identify several open challenges:
- Visual Reasoning Depth: Current models are still highly reliant on direct perception and struggle with abstract logic, spatial geometry, and complex chart/diagram parsing.
- Robustness and Bias: Data bias remains an issue; models still occasionally rely on statistical correlations within the training text rather than true visual grounding.
- Explainability: MLLMs function largely as black boxes. While attention maps and Chain-of-Thought rationales provide some insight, generating truly faithful, human-interpretable explanations of why a model chose an answer remains an open research problem.
- Natural Language Generation Flaws: Open-ended answer generation is prone to text hallucinations and formatting instability, which complicates objective evaluation.
Summary updated