Knowledge-representation-enhanced multimodal Transformer for scene text visual question answering

Yu Zhou, Jun Yu, Junjie Zhu, Zhenzhong KuangJournal of Image and Graphics 2022 · DOI

Thematic illustration of multimedia intelligence, representing the multimodal nature of the research.

Introduction

Standard Visual Question Answering (VQA) models excel at identifying visual objects but often struggle when a question requires reading and understanding text embedded within an image—a task known as Scene Text VQA. For instance, while a model might easily identify a "menu," it may fail to answer "What is the price of the burger?" without sophisticated text-reasoning capabilities.

Existing state-of-the-art models, such as the Multimodal Multi-Copy Mesh (M4C), typically treat visual objects and detected Optical Character Recognition (OCR) tokens as a flat sequence. This approach often overlooks two critical dimensions: spatial reasoning (the relative layout of text and objects) and semantic consistency (the linguistic relationship between image text and the answer vocabulary).

The KR-M4C Methodology

To address these limitations, researchers have proposed KR-M4C (Knowledge-Representation-Enhanced M4C). This model introduces a Knowledge-Representation-Enhanced Self-Attention (KRSA) module that integrates prior knowledge directly into the Transformer's attention mechanism.

Instead of relying solely on the model to learn relationships from scratch—which can lead to overfitting on smaller datasets—KR-M4C explicitly encodes two types of knowledge:

  1. Spatial Relation Knowledge ($R^{sp}$): The model calculates relative distances, angles, and size ratios between every pair of objects and OCR tokens. This allows the model to "understand" the physical layout of the scene.
  2. Semantic Relation Knowledge ($R^{se}$): Using pre-trained GloVe embeddings, the model assesses the semantic similarity between OCR tokens and the words in the answer vocabulary. This helps the model prioritize text that is contextually relevant to the question.

By baking these "priors" into the attention scores, the model is forced to weigh relationships based on physical and linguistic logic, rather than just statistical correlations in the training data.

Performance and Benchmarking

The KR-M4C model was evaluated on two primary benchmarks: TextVQA and ST-VQA.

Key Results

  • TextVQA: The model achieved a test accuracy of 42.99%, representing a 2.4% improvement over the baseline M4C. When augmented with additional training data, accuracy reached 49.63%.
  • ST-VQA: KR-M4C improved the Average Normalized Levenshtein Similarity (ANLS) score to 0.555, a 5% increase over previous spatially-aware models.

Importantly, these gains were achieved with minimal computational overhead. The model's parameter count increased by only 0.02M, and the FLOPs remained nearly identical to the baseline, demonstrating high efficiency.

Insights and Future Directions

The success of KR-M4C highlights that treating knowledge as a "bias" within the attention mechanism is more effective than simply adding it as an input feature. Spatial knowledge helps the model localize information, while semantic knowledge helps it select the correct answer.

Despite these advances, challenges remain. The model's performance is still dependent on the quality of the underlying OCR system. Future work will focus on developing end-to-end architectures that integrate OCR detection and VQA reasoning into a single trainable network to minimize error propagation and improve common-sense reasoning in complex scenes.

This summary is based on research published in the Journal of Image and Graphics.

Summary updated

Abstract

目的现有视觉问答方法通常只关注图像中的视觉物体,忽略了对图像中关键文本内容的理解,从而限制了图像内容理解的深度和精度。鉴于图像中隐含的文本信息对理解图像的重要性,学者提出了针对图像中场景文本理解的“场景文本视觉问答”任务以量化模型对场景文字的理解能力,并构建相应的基准评测数据集TextVQA(text visual question answering)和ST-VQA(scene text visual question answering)。本文聚焦场景文本视觉问答任务,针对现有基于自注意力模型的方法存在过拟合风险导致的性能瓶颈问题,提出一种融合知识表征的多模态Transformer的场景文本视觉问答方法,有效提升了模型的稳健性和准确性。方法对现有基线模型M4C(multimodal multi-copy mesh)进行改进,针对视觉对象间的“空间关联”和文本单词间的“语义关联”这两种互补的先验知识进行建模,并在此基础上设计了一种通用的知识表征增强注意力模块以实现对两种关系的统一编码表达,得到知识表征增强的KR-M4C(knowledge-representation-enhanced M4C)方法。结果在TextVQA和ST-VQA两个场景文本视觉问答基准评测集上,将本文KR-M4C方法与最新方法进行比较。本文方法在TextVQA数据集中,相比于对比方法中最好的结果,在不增加额外训练数据的情况下,测试集准确率提升2.4%,在增加ST-VQA数据集作为训练数据的情况下,测试集准确率提升1.1%;在ST-VQA数据集中,相比于对比方法中最好的结果,测试集的平均归一化Levenshtein相似度提升5%。同时,在TextVQA数据集中进行对比实验以验证两种先验知识的有效性,结果表明提出的KR-M4C模型提高了预测答案的准确率。结论本文提出的KR-M4C方法的性能在TextVQA和ST-VQA两个场景文本视觉问答基准评测集上均有显著提升,获得了在该任务上的最好结果。
BibTeX
@article{zhou2022knowledge,
  author = {Yu Zhou and Jun Yu and Junjie Zhu and Zhenzhong Kuang},
  title = {{Knowledge-representation-enhanced multimodal Transformer for scene text visual question answering}},
  journal = {Journal of Image and Graphics},
  year = {2022},
  doi = {10.11834/jig.211213},
}
Figure