When we look at an infographic, a meme, or a product label, our brains do not just process shapes and colors; we seamlessly read the text embedded within the image. However, conventional vision-language models exhibit a profound blind spot: they suffer from embedded visual text neglect.
Standard multimodal architectures are typically bimodal, designed to align global visual features (from a vision encoder) with an accompanying natural language caption (from a text encoder). While this works well for natural scenes—like matching a photo of a golden retriever with the word "dog"—it fails catastrophically when the critical semantic weight of an image is carried by the text baked into its pixels. To a standard vision transformer (ViT) or convolutional neural network (CNN), embedded text is merely a high-frequency texture or a geometric pattern, rather than a sequence of semantic tokens.
Overcoming this neglect requires shifting from a bimodal framework to architectures that explicitly treat scene text as a primary, distinct modality.
The Mechanism of Neglect: Why Pixels Aren't Tokens
To understand why standard models fail, consider how a vision encoder processes an image containing text. A ViT chops the image into patches and passes them through self-attention layers. While the model might learn to recognize the visual boundaries of a sign or a label, it lacks the character-level and language-modeling priors required to "read" the words.
As highlighted by Min Cao et al. in their survey on image-text retrieval, ignoring these textual cues discards highly descriptive data that is essential for cross-modal alignment. For instance, two storefronts might look visually identical in terms of architecture and lighting, but one has a sign reading "Coffee" and the other "Bakery." Without reading the text, a model cannot differentiate between them.
This limitation spans multiple domains:
- Digital Documents and Maps: Early work by Prasenjit Mitra et al. on Effectively Searching Maps in Web Documents noted that relying solely on external context like captions neglects the rich internal labels and legends embedded within the document's figures.
- Technical Literature: In scientific domains, Hiyori Yoshikawa et al. demonstrated that text-only analysis completely misses the primary summaries of complex procedures contained within embedded chemical diagrams and tables.
- Social Media: A vast amount of internet media relies on text-image interplay. Liang Zheng et al. introduced the Twitter100k dataset to reflect this messy reality, showing that ignoring embedded text in memes or screenshots discards the core semantic signal of the post.
The Trimodal Solution: Reading the Environment
To bridge this gap, researchers have evolved model architectures from bimodal (Image + Caption) to trimodal systems (Image + Caption + Scene Text). This is achieved by integrating Optical Character Recognition (OCR) directly into the model's perception pipeline.
Standard Bimodal Pipeline:
[Image] --------> Vision Encoder -----\
>---> [Fusion / Alignment]
[Caption] ------> Text Encoder -------/
Trimodal (Scene-Text Aware) Pipeline:
[Image] --------> Vision Encoder -----\
| \
v (OCR) >---> [Trimodal Fusion]
[Scene Text] ---> Text/OCR Encoder ---/
/
[Caption] ------> Text Encoder -------/
Andrés Mafla et al. formalized this approach with StacMR: Scene-Text Aware Cross-Modal Retrieval. They introduced a trimodal architecture called STARNet, which explicitly projects OCR-extracted scene text tokens into the same embedding space as the visual features and the caption. By treating scene text as its own modality, the model can leverage explicit semantic cues available in the environment that are otherwise invisible to the vision encoder.
However, simply extracting text via OCR and concatenating it with the caption is insufficient because it destroys the spatial context—the model loses track of where the text was written and what object it belongs to.
To solve this, Mengjun Cheng et al. developed ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval. ViSTA uses a transformer-based architecture with a specialized "fusion token." Instead of keeping the modalities separate until the final layer, ViSTA allows visual features and scene text tokens to interact dynamically through attention mechanisms. This prevents visually similar scenes from being misaligned by using the embedded text (such as brand names or dates) as a distinctive anchor.
Text-Aware Pre-Training
Rather than treating scene text as an afterthought or an auxiliary input, can we force models to learn to read during their foundational training phase?
Yijuan Lu addressed this with TAP: Text-Aware Pre-Training for Text-VQA and Text-Caption. TAP explicitly incorporates OCR-detected words into vision-language pre-training objectives. By masking scene text tokens and forcing the model to predict them based on both the surrounding visual context and the linguistic context, the model natively learns the alignment between pixel regions and text tokens. This pre-training paradigm yields massive accuracy boosts on tasks that require reading signs or labels.
Real-World Applications and Reasoning
When a model learns to read embedded text, its performance across complex, real-world multimodal tasks undergoes a qualitative shift.
1. Veracity and Intent (Fake News, Sarcasm, and Persuasion)
Deceptive content online rarely relies on text or images alone; it thrives on the friction or alignment between them. Peng Qi et al. developed the EM-FEND framework to improve fake news detection by explicitly modeling text complementation via OCR. They showed that multimodal misinformation frequently bakes deceptive text directly into images to evade text-only filters.
Similarly, Liqiang Jing et al. tackled multimodal sarcasm explanation through their TEAM framework, which extracts embedded text from memes to construct semantic graphs. Without this step, a model cannot detect the irony between an image and its message. In the realm of political or social discourse, Shaojun Yu et al. demonstrated with the SPLIT framework that the persuasive power and stance of an infographic reside almost entirely within its embedded text. While some verification frameworks, such as the one proposed by Wei-Yao Wang et al., attempt to use raw co-attention networks without explicit OCR, the consensus across the field emphasizes that extracting embedded text provides a critical safety net against missing key evidence.
2. E-Commerce and Personalization
In digital commerce, product specifications, dimensions, and ingredients are often trapped inside infographics or packaging images. Hejie Cui et al. introduced PV2TEA to "patch" this missing visual text into information extraction systems, capturing attributes that standard vision encoders overlook. On social platforms, digital stickers rely heavily on embedded text for humor and emotional expression. Heng Er Metilda Chee et al. leveraged fine-tuned vision-language models in their PerSRV framework to capture these text-based punchlines, moving sticker retrieval beyond simple visual similarity to deep emotional alignment.
3. Assistive Technology and Question Answering
For visually impaired users, knowing that an object is a "can" is rarely enough—they need to know if it is a can of soup or a can of soda. Pierre Dognin et al., in their analysis of the VizWiz challenge, demonstrated that goal-oriented captioning requires rotation-invariant OCR and a specialized "copy mechanism" to inject text directly from the image into the output caption.
To evaluate these capabilities systematically, Ali Furkan Biten et al. established the ST-VQA benchmark. This dataset forces models to answer questions that can only be solved by reading scene text, exposing the limitations of traditional VQA models and establishing metrics that account for real-world OCR imperfections.
4. Mitigating Hallucinations in Logical Reasoning
Even modern Multimodal Large Language Models (MLLMs) frequently hallucinate when faced with scientific or mathematical diagrams because they fail to ground their reasoning in the text embedded within the figures. To combat this, Mingkang Chen et al. introduced VTPerception-R1. This framework enforces a two-stage reasoning process: the model must explicitly perceive and describe the embedded visual and textual tokens before it is allowed to generate a solution. By rewarding consistency between what the model explicitly reads and how it reasons, they significantly reduced logical errors in technical tasks.
Conclusion
Embedded visual text neglect highlights a fundamental truth in multimodal AI: perception is not just about seeing objects, but also about reading context. By evolving from rigid bimodal architectures to trimodal, text-aware frameworks, modern models can finally bridge the semantic gap between pixels and language, allowing them to navigate a world where text and image are inextricably linked.
Key Foundational Work
Latent Factors of Visual Popularity Prediction (2015) — contributed 1 concept elements to this direction.
Effectively Searching Maps in Web Documents (2009) — contributed 1 concept elements to this direction.