VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning

Kang Chen, Xiangqian WuCVPR 2024 · DOI · arXiv

Abstract

Researchers from the Harbin Institute of Technology have introduced VTQA, a new challenge and dataset designed to push AI beyond simple image recognition. By requiring models to perform multi-hop reasoning across both long-form text and complex images, VTQA forces a deeper level of entity alignment and cross-media understanding than previous benchmarks.

Introduction: The Need for Cross-Media Reasoning

While Visual Question Answering (VQA) has been a prominent research area, existing models often struggle with tasks requiring deep integration of visual and textual information. Many current datasets allow models to bypass true reasoning by relying solely on visual cues or simple text matching. VTQA addresses these gaps by providing a dataset where questions cannot be answered by the image or the text in isolation. To succeed, a model must find an entity in the text, align it with a specific object in the image, and perform multi-hop reasoning to derive the answer.

An example from the VTQA dataset illustrating the necessity of aligning visual regions with long-form textual descriptions to answer complex questions.

The VTQA Dataset

The VTQA dataset consists of 10,124 image-text pairs and 23,781 questions. Unlike typical VQA datasets that use short captions, VTQA features human-annotated descriptive paragraphs ranging from 300 to 600 words. This complexity moves the field toward "Document VQA," where information retrieval is as critical as visual processing.

Questions in VTQA are categorized into three types:

  • YN (Yes/No): Binary verification tasks.
  • E (Extracted): Answers found directly in the text.
  • G (Generated): Answers requiring complex synthesis of both image and text.

Methodology: The KECMRN Architecture

To establish a baseline, the authors propose the Key Entity Cross-Media Reasoning Network (KECMRN). The architecture utilizes two specialized modular layers built on top of standard LSTM and Faster R-CNN backbones:

  1. Key Entity Extract (KEE) Layer: Employs attention mechanisms to filter noise and identify relevant entities in both media based on the question.
  2. Cross-Media Reason (CMR) Layer: Facilitates multi-hop reasoning by passing information between visual and textual representations through self-attention and global cross-attention. This allows the model to update features based on cross-media context before generating an answer.

Key Insights and Results

The core challenge of VTQA is entity alignment. For instance, to answer a question about "Elena," the model must first identify which person in the image corresponds to the textual description of "Elena" (e.g., "blond hair") before performing further reasoning.

The KECMRN baseline achieved an Exact Match (EM) score of 51.3% on the test set. While performance on Yes/No questions was strong (77.6%), the 'Generated' (G) category proved significantly more difficult, with an F1 score of only 0.511. This gap highlights the substantial room for improvement in cross-media fusion and creative synthesis.

Conclusion

VTQA represents a significant step toward more human-like multi-modal intelligence. By emphasizing entity alignment and long-form text comprehension, it challenges researchers to develop models that truly "see" and "read" in tandem. Future work may involve integrating Large Language Models (LLMs) into the KECMRN framework to improve the naturalness and accuracy of open-ended answers.

Summary updated

BibTeX
@inproceedings{chen2024vtqa,
  author = {Kang Chen and Xiangqian Wu},
  title = {{VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02570},
}
Figure