Abstract
Outside-knowledge visual question answering (OK-VQA) requires an agent to comprehend an image, retrieve relevant knowledge from the web, and synthesize this information to provide an answer. Traditional approaches often fuse image and question features in a multi-modal space, which can be inflexible for integrating vast external knowledge bases. We propose a new paradigm: the Transform-Retrieve-Generate (TRiG) framework. TRiG transforms images into plain text, enabling knowledge retrieval and generative answering entirely within the natural language space. This approach leverages the scale of massive knowledge bases and the reasoning capabilities of pretrained language models. Experimental results demonstrate that TRiG outperforms existing supervised state-of-the-art methods by an absolute margin of at least 11.1%.
Introduction: Moving Beyond the Multi-Modal "Squeeze"
Visual Question Answering (VQA) has evolved significantly, yet "Outside-Knowledge" VQA (OK-VQA) remains a complex challenge. While standard VQA might ask a model to identify a cat, OK-VQA might ask for the specific historical origin of a breed shown in an image.
Most existing OK-VQA models rely on a "fusion" strategy, mashing image and question embeddings into a joint mathematical space. However, when integrating external knowledge—such as the 21 million passages in Wikipedia—this multi-modal bottleneck often "squeezes" out rich textual information, leading to loss of detail and inflexibility. Researchers from UCLA and Amazon argue that the natural language space is a more effective environment for knowledge-heavy reasoning, allowing models to "read" the world rather than just "see" it.
The TRiG Framework
The TRiG framework (Transform-Retrieve-Generate) shifts the problem into the natural language domain through a three-stage pipeline:
- Transform (Image-to-Text): To minimize information loss, the image is converted into three textual formats: global captions, dense labels (detailed tags for objects and attributes like "rusty brown truck"), and OCR (text found within the image).
- Retrieve (Dense Passage Retrieval): Using the question and generated captions as a query, the system employs Dense Passage Retrieval (DPR) to scan 21 million Wikipedia passages, pulling the top k most relevant snippets.
- Generate (Multi-Passage QA): A T5 (Text-to-Text Transfer Transformer) model processes the question, visual context, and retrieved passages. Using a "Fusion-in-Decoder" approach, it synthesizes information across all passages to generate a free-form answer.
Key Insights and Results
A central finding of this research is that language can serve as a universal interface for multi-modal tasks. By treating OK-VQA as an Open-Domain QA problem, TRiG can utilize "plug-and-play" components from the NLP field.
Performance and Efficiency
TRiG achieved a VQA score of 50.50%, significantly surpassing the previous supervised state-of-the-art (MAVEx at 39.40%). Notably, TRiG outperformed PICa—a model utilizing the 175B-parameter GPT-3—by 2.5%, despite TRiG’s T5-large backbone having only 0.77B parameters. This suggests that a specialized retrieval pipeline is more effective than sheer model scale.
The Role of Retrieval Scale
The effectiveness of the framework is closely tied to the number of retrieved passages (k). As shown in the experimental analysis, both the VQA score and the retrieval "Hit@k" (the probability that the correct answer is contained within the retrieved passages) improve as k increases toward 100.
Limitations and Future Directions
While TRiG sets a new benchmark, two primary bottlenecks remain. Approximately 50% of errors stem from the "Transform" stage failing to capture nuanced visual details (e.g., specific biological features of an animal). Another 24% of errors are due to retrieval failures where the "smoking gun" passage is not found. Future work will focus on optimizing retrievers specifically for OK-VQA and improving image-to-text descriptions to capture more granular visual data.
By moving the reasoning process into the natural language space, TRiG demonstrates that the most powerful way to solve complex visual tasks may be to leverage the vast, structured knowledge already encoded in human language.
Summary updated