TextCaps: a Dataset for Image Captioning with Reading Comprehension

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh

Abstract

Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.

Introduction: The Need for Reading Comprehension in Captioning

While AI has made massive strides in describing objects (e.g., "a dog on a rug") and Optical Character Recognition (OCR) has matured, these two fields rarely talk to each other. Standard captioning datasets like COCO largely ignore text; in fact, only about 2.7% of COCO captions even mention text present in the image.

However, for a visually impaired person, knowing there is "a red sign" is often useless compared to knowing it is a sign for "Mornington Crescent Station." Current models treat text as an afterthought or fail to reason about it entirely. The authors address this gap by introducing a task where reading and understanding text is critical to generating a meaningful description.

Comparison of standard model captions vs. human ground truth on images containing text.

The TextCaps Dataset

The star of the paper is the TextCaps dataset, which contains 142,040 captions for 28,408 images. These images were specifically filtered from the Open Images dataset to ensure they contain text. Unlike TextVQA, where "answers" are typically 1-2 words, TextCaps requires full, grammatically correct sentences that integrate text tokens naturally.

Methods: M4C-Captioner

The authors propose M4C-Captioner, an adaptation of the Multimodal Multi-Copy Mesh (M4C) architecture originally designed for Visual Question Answering (VQA).

  • Multimodal Fusion: The model uses a transformer-based architecture to fuse three distinct inputs: visual objects (detected via Faster R-CNN), OCR tokens (extracted via the Rosetta system), and the previously generated words in the caption.
  • The Pointer Network: Unlike standard captioners that only pick words from a fixed vocabulary, M4C-Captioner uses a dynamic pointer network. This allows the model to "switch" on the fly between generating a common word (like "the" or "bottle") and "copying" a specific string of text found in the image (like "Coca-Cola").
  • Rich OCR Features: To help the model understand the text, each OCR token is represented by its semantic meaning (FastText/PHOC embeddings), its visual appearance, and its spatial location (bounding box coordinates).

Key Insights and Reasoning

A major challenge is the "switching" logic. A model might need to switch between its internal vocabulary and the image's OCR tokens multiple times in a single sentence (e.g., "A [sign] that [says] [‘STOP’] [on] [a] [red] [post]").

Furthermore, the dataset isn't just about transcription. Captions often require inference. For example, seeing numbers on a scoreboard requires the model to understand the context of those numbers to generate a meaningful summary.

A scoreboard example where reasoning about text (scores, quarter, down) is necessary for a complete description.

Spatial information (where the text is located) is one of the most critical features for high-quality captions, as it helps the model associate text with specific objects. Additionally, many text tokens in the test set (like specific brand names) never appeared in the training set, forcing models to actually read the pixels rather than just memorizing common words.

Results and Performance Gap

  • Standard Models Fail: State-of-the-art models like BUTD and AoANet struggle significantly on TextCaps because they lack a mechanism to "copy" text from the image, scoring around 33-34 CIDEr.
  • M4C-Captioner Dominates: The proposed M4C-Captioner achieved a CIDEr score of 81.0 on the test set.
  • The Human Gap: Despite the success of M4C, there is still a massive gap. Humans achieve a CIDEr score of 125.5. This indicates that while models are getting better at copying text, they still lack the deep semantic reasoning humans use to summarize and interpret what they read.
  • Ground Truth Boost: When the model was given "perfect" human-annotated OCR instead of machine-generated OCR, its score jumped to 97.2, suggesting that improvements in basic OCR technology will directly translate to better captioning.

Conclusion

TextCaps serves as a catalyst for the community to develop models that can truly "see and read" the world simultaneously. While M4C-Captioner makes significant progress, future work must address complex paraphrasing and dense, stylized text to bridge the gap to human-level performance.

Summary updated

Abstract

Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.
BibTeX
@inproceedings{sidorov2020textcaps,
  author = {Oleksii Sidorov and Ronghang Hu and Marcus Rohrbach and Amanpreet Singh},
  title = {{TextCaps: a Dataset for Image Captioning with Reading Comprehension}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=nBV9b5GAzrU},
}
Figure