Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model

Yixuan Qiao, Hao Chen, Jun Wang, Yihao Chen, Xianbin Ye, Ziliang Li, Xianbiao Qi, Peng Gao, Guotong Xie

Abstract

TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only.

Introduction

Standard Visual Question Answering (VQA) focuses on understanding objects and their spatial relationships. However, TextVQA introduces a significant layer of complexity: models must "read" and reason about text found within the scene, such as street signs, brand names, or price tags. This task requires the seamless fusion of three distinct information types: natural language questions, visual features of objects and backgrounds, and scene text detected via Optical Character Recognition (OCR).

Existing models often struggle to align text with its physical context. Team Mia addressed this by treating text as a primary visual modality, leveraging the massive T5-3B sequence-to-sequence model to bridge the gap between machine performance and human-level reasoning.

Methodology

The core of Team Mia's approach is the T5-3B (Text-to-Text Transfer Transformer) architecture. Unlike traditional classification-based VQA models, T5 is a generative model that produces answers token-by-token, offering greater flexibility in vocabulary.

Key Innovations

  • Relative Position Prediction (RPP): In addition to standard Masked Language Modeling (MLM), the team introduced RPP during pre-training. This task requires the model to predict the spatial distance between scene text and visual objects, forcing it to learn the physical relationships between labels and their corresponding objects.
  • Multi-modal Input Fusion: The T5 encoder processes a "mega-sequence" containing question text, object labels, scene text labels, and their respective visual features (extracted using VinVL and the Microsoft OCR API).
  • Embedding-level Adversarial Training: To enhance robustness, adversarial perturbations were added directly to the mathematical embeddings of each modality. This method provides the benefits of adversarial training without the high computational cost of backpropagating through a vision backbone.
  • Fuzzy Post-processing: The fuzzywuzzy library was employed to perform string matching, correcting minor OCR or generation errors in the final output.

Datasets and Pre-training

The model underwent a two-stage training process. First, it was pre-trained on a massive dataset of 9 million scene text images, comprising both real-world and synthetically generated data. Subsequently, the model was fine-tuned on the TextVQA dataset.

Statistics of the TextVQA dataset used for fine-tuning.

As shown in the table above, the TextVQA dataset is highly specialized, with 100% of images containing text and nearly 40% of answers being directly extractable from OCR tokens.

Results and Discussion

Team Mia secured first place in the TextVQA Challenge 2021 with a score of 73.67%. This performance represents a substantial leap over previous years, nearly closing the gap with the human baseline.

Comparison of model performance on the TextVQA Challenge 2021.

The results highlight several key insights:

  • Scale and Common Sense: The 3-billion parameter T5 model provides a strong foundation of "common sense" reasoning that aids in interpreting visual data.
  • Generative Flexibility: The seq2seq approach allows the model to either copy text from the image or generate contextually appropriate words.
  • Spatial Awareness: The success of the RPP task confirms that spatial context is critical for answering complex visual questions.

Conclusion

While Team Mia's approach significantly advanced the state-of-the-art, a ~13% gap remains compared to human performance. Future work will focus on improving reasoning for multi-word answers and developing more sophisticated "hard-negative" mining during pre-training to further sharpen the model's discriminative capabilities.

Summary updated

Abstract

TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only.
BibTeX
@misc{qiao2021winner,
  author = {Yixuan Qiao and Hao Chen and Jun Wang and Yihao Chen and Xianbin Ye and Ziliang Li and Xianbiao Qi and Peng Gao and Guotong Xie},
  title = {{Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model}},
  year = {2021},
  eprint = {2106.15332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.15332},
}
Figure