Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering

Feng Gao, Qing Ping, Govind Thattai, Aishwarya N. Reganti, Ying Nian Wu, Prem NatarajanCVPR 2022 · OpenReview

Abstract

Outside-knowledge visual question answering (OK-VQA) requires an agent to comprehend an image, retrieve relevant knowledge from the web, and synthesize this information to provide an answer. Traditional approaches often fuse image and question features in a multi-modal space, which can be inflexible for integrating vast external knowledge bases. We propose a new paradigm: the Transform-Retrieve-Generate (TRiG) framework. TRiG transforms images into plain text, enabling knowledge retrieval and generative answering entirely within the natural language space. This approach leverages the scale of massive knowledge bases and the reasoning capabilities of pretrained language models. Experimental results demonstrate that TRiG outperforms existing supervised state-of-the-art methods by an absolute margin of at least 11.1%.

Introduction: Moving Beyond the Multi-Modal "Squeeze"

Visual Question Answering (VQA) has evolved significantly, yet "Outside-Knowledge" VQA (OK-VQA) remains a complex challenge. While standard VQA might ask a model to identify a cat, OK-VQA might ask for the specific historical origin of a breed shown in an image.

Most existing OK-VQA models rely on a "fusion" strategy, mashing image and question embeddings into a joint mathematical space. However, when integrating external knowledge—such as the 21 million passages in Wikipedia—this multi-modal bottleneck often "squeezes" out rich textual information, leading to loss of detail and inflexibility. Researchers from UCLA and Amazon argue that the natural language space is a more effective environment for knowledge-heavy reasoning, allowing models to "read" the world rather than just "see" it.

The TRiG Framework

The TRiG framework (Transform-Retrieve-Generate) shifts the problem into the natural language domain through a three-stage pipeline:

  1. Transform (Image-to-Text): To minimize information loss, the image is converted into three textual formats: global captions, dense labels (detailed tags for objects and attributes like "rusty brown truck"), and OCR (text found within the image).
  2. Retrieve (Dense Passage Retrieval): Using the question and generated captions as a query, the system employs Dense Passage Retrieval (DPR) to scan 21 million Wikipedia passages, pulling the top k most relevant snippets.
  3. Generate (Multi-Passage QA): A T5 (Text-to-Text Transfer Transformer) model processes the question, visual context, and retrieved passages. Using a "Fusion-in-Decoder" approach, it synthesizes information across all passages to generate a free-form answer.
The TRiG framework architecture, illustrating the Transform, Retrieve, and Generate stages.

Key Insights and Results

A central finding of this research is that language can serve as a universal interface for multi-modal tasks. By treating OK-VQA as an Open-Domain QA problem, TRiG can utilize "plug-and-play" components from the NLP field.

Performance and Efficiency

TRiG achieved a VQA score of 50.50%, significantly surpassing the previous supervised state-of-the-art (MAVEx at 39.40%). Notably, TRiG outperformed PICa—a model utilizing the 175B-parameter GPT-3—by 2.5%, despite TRiG’s T5-large backbone having only 0.77B parameters. This suggests that a specialized retrieval pipeline is more effective than sheer model scale.

The Role of Retrieval Scale

The effectiveness of the framework is closely tied to the number of retrieved passages (k). As shown in the experimental analysis, both the VQA score and the retrieval "Hit@k" (the probability that the correct answer is contained within the retrieved passages) improve as k increases toward 100.

Performance metrics (VQA/EM scores and Hit@k) as the number of retrieved passages (K) increases.

Limitations and Future Directions

While TRiG sets a new benchmark, two primary bottlenecks remain. Approximately 50% of errors stem from the "Transform" stage failing to capture nuanced visual details (e.g., specific biological features of an animal). Another 24% of errors are due to retrieval failures where the "smoking gun" passage is not found. Future work will focus on optimizing retrievers specifically for OK-VQA and improving image-to-text descriptions to capture more granular visual data.

By moving the reasoning process into the natural language space, TRiG demonstrates that the most powerful way to solve complex visual tasks may be to leverage the vast, structured knowledge already encoded in human language.

Summary updated

Abstract

Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge. In this paper, we call for an alternative paradigm for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the natural language space. This paradigm takes advantage of the sheer volume of gigantic knowledge bases and the richness of pretrained language models. A Transform-Retrieve-Generate framework (TRiG) framework is proposed <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The code of this work will be made public., which can be plug-and-played with alternative image-to-text models and textual knowledge bases. Experimental results show that our TRiG framework outperforms all state-of-the-art supervised methods by at least 11.1 % absolute margin.
BibTeX
@inproceedings{gao2022transform,
  author = {Feng Gao and Qing Ping and Govind Thattai and Aishwarya N. Reganti and Ying Nian Wu and Prem Natarajan},
  title = {{Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=NVC8uPZLwAt},
}
Figure