TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation

Jun Wang, Mingfei Gao, Yuqian Hu, Ramprasaath R. Selvaraju, Chetan Ramaiah, Ran Xu, Joseph F. JáJá, Larry S. Davis

Abstract

Text-based Visual Question Answering (Text-VQA) requires models to reason about text within images. Despite the abundance of scene text in datasets, human annotations are often sparse, leaving significant amounts of "dark data" unutilized. We present TAG (Text-aware visual question-answer Generation), a framework designed to automatically generate high-quality QA pairs from this unused text. By leveraging a multimodal transformer and a dynamic pointer network, TAG creates diverse training samples that significantly enhance the performance of downstream Text-VQA models. Our experiments on TextVQA and ST-VQA benchmarks demonstrate that TAG-augmented data allows models to achieve state-of-the-art results without additional human labeling.

1. Introduction

Visual Question Answering (VQA) has seen rapid progress, yet the sub-task of Text-VQA remains particularly challenging. Models must not only recognize objects but also read and interpret text in various fonts, orientations, and contexts. A major bottleneck in this field is the scarcity of annotated data. While a single image may contain dozens of text tokens, human annotators typically provide only one or two question-answer pairs. This leaves the majority of scene text as "dark data"—present in the image but ignored during the training process.

To address this, we introduce TAG, a system that converts underutilized scene text into "training gold." By automatically generating questions for which the scene text serves as the answer, TAG effectively scales training datasets without the cost of human intervention.

2. The TAG Framework

The core of our approach is the TAG module, which reverses the standard VQA task: instead of answering a question, it generates a question based on a specific text "answer" found in the image.

2.1 Input Modalities

TAG employs a multimodal transformer to fuse three distinct types of information:

  • Extended Answer Words: The target text token selected as the answer, enriched with spatial and semantic context from an object detector.
  • Visual Objects: Visual features extracted using a Faster R-CNN detector to provide physical layout and environmental context.
  • OCR Tokens: All text detected within the scene, typically extracted via high-quality OCR engines like Microsoft-OCR.

2.2 Architecture and Generation

The architecture utilizes a multi-layer transformer to model the relationships between words and their visual surroundings. For instance, it learns to associate the word "Saigon" with the visual features of a beer label.

Question generation is performed by an auto-regressive decoder equipped with a dynamic pointer network. This network allows the model to either generate common linguistic tokens (e.g., "what", "is") or "point" to specific OCR tokens within the image to incorporate them directly into the generated question.

3. Data Augmentation Strategy

A critical component of TAG is the strategy for selecting which text tokens to use as answers. During the augmentation phase, the system employs a "largest box" heuristic. By selecting the text with the largest bounding box, the model prioritizes text that is likely to be semantically central to the image and less susceptible to OCR noise. Once an answer is selected, TAG generates a corresponding question, and the new pair is added to the training set for downstream models such as M4C or TAP.

4. Experimental Evaluation

We evaluated the impact of TAG-generated data on two primary benchmarks:

  • TextVQA: Approximately 28,000 images from Open Images featuring human-annotated questions.
  • ST-VQA (Scene Text VQA): Approximately 23,000 images sourced from ICDAR and ImageNet.

TAG was used to augment these datasets, and the resulting data was used to train state-of-the-art Text-VQA models.

5. Results and Insights

Our findings suggest that increasing data volume through automated generation can be more effective than complex pre-training on massive external datasets.

  • Performance Gains: Integrating TAG-generated data improved the accuracy of the TAP model on the TextVQA dataset from 49.91% to 52.54% in constrained settings.
  • Efficiency: TAG-augmented models outperformed competitors that utilized significantly larger external datasets (e.g., 1.4 million samples in the ST-VQA unconstrained setting), demonstrating the high quality of the generated pairs.
  • Model-Agnostic Nature: Because TAG functions as a data augmentation tool, its benefits are disentangled from specific model architectures. It can be applied to improve any existing or future Text-VQA model.

Qualitative analysis shows that TAG generates logical and contextually relevant questions. For example, given an image of a "Saigon" beer bottle, TAG might generate "What is the brand of the wine?", providing useful training signal even if the category classification (wine vs. beer) is slightly off.

6. Conclusion and Future Work

TAG demonstrates that scene text is a vastly untapped resource for training Text-VQA systems. By converting "dark data" into structured QA pairs, we can significantly boost model performance without additional human labor.

Future research will focus on more sophisticated answer selection strategies, moving beyond the "largest box" heuristic to identify more "interesting" or semantically complex text. Additionally, improving the robustness of the system to OCR errors through tighter integration between detection and generation remains a key area for development.

Summary updated

Abstract

Text-VQA aims at answering questions that require understanding the textual cues in an image. Despite the great progress of existing Text-VQA methods, their performance suffers from insufficient human-labeled question-answer (QA) pairs. However, we observe that, in general, the scene text is not fully exploited in the existing datasets -- only a small portion of the text in each image participates in the annotated QA activities. This results in a huge waste of useful information. To address this deficiency, we develop a new method to generate high-quality and diverse QA pairs by explicitly utilizing the existing rich text available in the scene context of each image. Specifically, we propose, TAG, a text-aware visual question-answer generation architecture that learns to produce meaningful, and accurate QA samples using a multimodal transformer. The architecture exploits underexplored scene text information and enhances scene understanding of Text-VQA models by combining the generated QA pairs with the initial training data. Extensive experimental results on two well-known Text-VQA benchmarks (TextVQA and ST-VQA) demonstrate that our proposed TAG effectively enlarges the training data that helps improve the Text-VQA performance without extra labeling effort. Moreover, our model outperforms state-of-the-art approaches that are pre-trained with extra large-scale data. Code is available at https://github.com/HenryJunW/TAG.
BibTeX
@inproceedings{wang2022tag,
  author = {Jun Wang and Mingfei Gao and Yuqian Hu and Ramprasaath R. Selvaraju and Chetan Ramaiah and Ran Xu and Joseph F. JáJá and Larry S. Davis},
  title = {{TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation}},
  booktitle = {BMVC},
  year = {2022},
  url = {https://openreview.net/forum?id=Sr_mM8us4Rs},
}
Figure