Abstract
Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.
Introduction
Traditional image captioning models have become remarkably proficient at identifying objects, such as "a person holding a phone" or "a red sign on a wall." However, these models are effectively "illiterate" in man-made environments where the text on objects is often more critical than the objects themselves. For instance, knowing a sign is red does not assist in navigation unless the model can read that it says "Mornington Crescent Station."
Current datasets like COCO are insufficient for training models to read; only about 2.7% of COCO captions mention text found in the image. For AI to truly assist visually impaired users or navigate human spaces, it must integrate Optical Character Recognition (OCR) with visual context to generate meaningful descriptions.
Methodology: M4C-Captioner
To address this challenge, researchers adapted the Multimodal Multi-Copy Mesh (M4C) architecture, originally designed for Visual Question Answering (VQA), into a new baseline called M4C-Captioner.
Key technical features include:
- Multimodal Transformer: A rich embedding space that fuses visual features (from Faster R-CNN), spatial features (bounding box coordinates), and semantic features (FastText and PHOC) of OCR tokens.
- Dynamic Pointer Network: Unlike standard captioners that only pick words from a fixed vocabulary, this model can "point" to and copy specific OCR tokens detected in the image.
- Iterative Decoding: The model generates captions word-by-word, deciding at each step whether to use a common word (like "the" or "is") or an OCR token (like "Pepsi" or "12:30").
The TextCaps Dataset
The TextCaps dataset consists of 142,040 human-written captions for 28,408 images sourced from the Open Images dataset. Unlike VQA datasets where answers are typically short, TextCaps requires full, grammatically correct sentences (averaging 12.4 words) that weave together visual descriptions and text.
The dataset introduces several unique challenges:
- The "Switching" Challenge: Models must frequently switch between internal vocabulary and OCR tokens. Most TextCaps captions require two or more "switches" per sentence.
- Zero-Shot Reading: 45% of unique OCR tokens in the test set never appeared in the training set, forcing models to rely on reading and copying rather than memorization.
- Reasoning vs. Copying: Humans often paraphrase or infer meaning. For example, a scoreboard reading "Rice 18, ECU 17" implies "Rice is winning," a level of reasoning current models struggle to achieve.
Results and Performance Gap
The results highlight a significant gap between current state-of-the-art models and human performance:
- M4C-Captioner achieved a CIDEr score of 81.0 on the test set, outperforming standard models like BUTD and AoANet.
- Human Performance sits at a CIDEr score of 125.5, representing a massive performance gap.
- OCR Bottleneck: When provided with "ground-truth" OCR (perfect text recognition), the model's score jumped to 97.2, suggesting that current OCR technology is a primary bottleneck.
Conclusion
While TextCaps provides a path forward for image captioning with reading comprehension, several hurdles remain. Models still struggle with indirect reasoning—such as inferring that a "5:43p" timestamp refers to a train departure—and are sensitive to OCR failures. Future research must focus on tighter integration between OCR detection and language generation to bridge the gap toward human-level scene understanding.
Summary updated