Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

Jiayi Kuang, Jingyou Xie, Haohao Luo, Ronghao Li, Zhe Xu, Xianfeng Cheng, Yinghui Li, Xika Lin, Ying Shen

TL;DR

This survey paper provides a comprehensive, up-to-date overview of the Visual Question Answering (VQA) landscape, charting its evolution from conventional deep learning architectures to modern Multimodal Large Language Models (MLLMs). By systematically classifying the field into perceptual (understanding) and cognitive (inference) dimensions, the authors highlight how VQA has transitioned from simple feature matching to sophisticated multi-hop reasoning and instruction tuning.

What Problem Does This Paper Address?

Visual Question Answering (VQA) sits at the challenging intersection of Computer Vision (CV) and Natural Language Processing (NLP). The core task is simple to state—predict an accurate natural language answer given an image (or video) and a related question—but incredibly difficult to execute.

Early VQA systems focused primarily on basic perception: extracting visual features from an image, textual features from a question, and fusing them to predict an answer. However, real-world VQA requires more than just "seeing"; it demands cognitive reasoning, context awareness, and often the integration of external common-sense knowledge. With the recent explosion of Multimodal Large Language Models (MLLMs), the paradigm of VQA has fundamentally shifted. This paper addresses the need for a unified, timely taxonomy that bridges traditional deep learning fusion strategies with the latest zero-shot, few-shot, and generative reasoning capabilities of MLLMs.

Fig. 1. The data used in VQA tasks, with the conclusion of the understanding and inference methods from conventional models to multimodal large language models.

Methods

The authors organize the VQA framework into a clear taxonomy divided into two primary phases: Natural Language Understanding of Image and Text and Natural Language Inference.

Fig. 2. Taxonomy Graph of VQA Task.

1. Multimodal Understanding (Perception)

The paper outlines the progression of feature extraction and fusion:

  • Conventional Deep Learning: Early approaches relied on CNNs (like VGG or ResNet) for images and RNNs/LSTMs for text, combining them via simple vector operations or bilinear pooling. This evolved into attention mechanisms (co-attention) and Graph Neural Networks (GNNs) to model spatial and semantic relationships.
  • Vision-Language Pre-training (VLP): Models like ViLBERT and LXMERT introduced dual-stream architectures, while UNITER and ViLT pioneered single-stream architectures to learn joint representations on massive unlabeled datasets.
  • MLLM Paradigms: The survey identifies three main ways modern MLLMs comprehend visual data:
  • LLM-Aided Visual Understanding: The LLM acts as a central controller, breaking down the question and dispatching tasks to external vision tools (e.g., Visual ChatGPT).
  • Image-to-Text Generation: Transforming the image into captions or OCR text before feeding it directly into a text-only LLM (e.g., PICa).
  • General Multimodal LLMs: End-to-end networks where visual features are directly mapped into the LLM's textual space using alignment modules like Q-Former or Perceiver Resampler (e.g., BLIP-2, LLaVA).

2. Natural Language Inference (Cognition)

For reasoning and answer generation, the methods span:

  • Knowledge-Based Reasoning: Extracting internal or external knowledge (from sources like DBpedia or ConceptNet) and performing one-hop or multi-hop reasoning using memory networks or knowledge graphs.
  • MLLM Reasoning: Leveraging advanced prompting techniques such as multimodal instruction tuning, in-context learning (ICL), and Multimodal Chain-of-Thought (M-CoT) to generate step-by-step rationales before arriving at an answer.

Datasets

The survey categorizes VQA benchmarks based on their reliance on external knowledge:

  • Datasets without External Knowledge: Classic benchmarks like DAQUAR, VQAv1, VQAv2, CLEVR (synthetic geometry), GQA, and specialized tasks like DocVQA and VizWiz.
  • Datasets with External Knowledge Base: Benchmarks explicitly requiring outside information, including KB-VQA, FVQA, OK-VQA, and KVQA.
  • MLLM Benchmarks: The paper highlights a new breed of comprehensive evaluation suites like MME, SEED-BENCH, MathVista, and MMMU, which assess MLLMs across highly specialized domains (e.g., mathematics, physics, and logic).

Key Insights and Ideas

  • From Perception to Cognition: VQA has evolved from a "what you see is what you get" task into a complex cognitive challenge. The most successful modern architectures focus heavily on how knowledge is retrieved and structured for multi-step deduction.
  • The Power of Frozen LLMs: A key architectural trend in MLLMs is keeping the heavy language backbone frozen while training lightweight alignment modules (like adapters or Q-Formers). This drastically lowers computational costs while inheriting the LLM's powerful text-reasoning capabilities.
  • Shortcuts vs. Grounding: Traditional deep learning models frequently suffered from "language bias" (e.g., answering "yellow" to any question starting with "What color is the banana?"). While MLLMs mitigate this through better cross-modal grounding, they introduce new risks, such as hallucinating objects not present in the scene.

Key Results

The paper provides a comprehensive performance meta-analysis across four benchmark datasets (VQAv2, OK-VQA, GQA, and VizWiz):

  • VQAv2: MLLMs achieve state-of-the-art results, with mPLUG-Huge leading at 83.62% accuracy, compared to early deep learning baselines like MUTAN at 67.4%.
  • OK-VQA (Knowledge-Intensive): PaLI-X achieves the highest accuracy at 66.1%, significantly outperforming specialized graph-based reasoning models like Mucko (29.2%), proving that massive scale and unified pre-training can implicitly capture vast amounts of external knowledge.
  • VizWiz: Models like PaLI achieve up to 73.3% accuracy on real-world, low-quality images taken by visually impaired users.

Limitations and Future Work

Despite massive progress, the authors identify several open challenges:

  • Visual Reasoning Depth: Current models are still highly reliant on direct perception and struggle with abstract logic, spatial geometry, and complex chart/diagram parsing.
  • Robustness and Bias: Data bias remains an issue; models still occasionally rely on statistical correlations within the training text rather than true visual grounding.
  • Explainability: MLLMs function largely as black boxes. While attention maps and Chain-of-Thought rationales provide some insight, generating truly faithful, human-interpretable explanations of why a model chose an answer remains an open research problem.
  • Natural Language Generation Flaws: Open-ended answer generation is prone to text hallucinations and formatting instability, which complicates objective evaluation.

Summary updated

Abstract

Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.
BibTeX
@article{kuang2025natural,
  author = {Jiayi Kuang and Jingyou Xie and Haohao Luo and Ronghao Li and Zhe Xu and Xianfeng Cheng and Yinghui Li and Xika Lin and Ying Shen},
  title = {{Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey}},
  journal = {ACM Computing Surveys},
  year = {2025},
  doi = {10.1145/3711680},
}
Figure