Explicit Text Extraction for Multimodal Transformers

71 concepts60 papers1 proposal

Modern Large Multimodal Models (LMMs) excel at holistic scene understanding, yet they frequently stumble when faced with a seemingly simple task: reading. When a model processes an image, its visual encoder (typically a Vision Transformer) tokenizes the image into a grid of fixed-size patches. While this downsampling works beautifully for identifying macro-level objects like trees or faces, it introduces a severe resolution bottleneck for fine-grained textual information. Small fonts, dense chart labels, and stylized text are effectively compressed into unreadable pixel noise.

To circumvent this limitation, a prominent research direction has emerged: explicit text extraction fusion. Instead of forcing the visual encoder to parse text from raw pixels, these architectures use an external Optical Character Recognition (OCR) engine or a specialized text extraction pipeline to convert embedded pixels directly into discrete semantic text tokens. By feeding these tokens directly into the language backbone, the model bypasses the visual resolution bottleneck entirely.


Dual-Stream and Tri-Stream Architectures

How do we feed this extracted text back into a multimodal model? The most intuitive approach is to treat the OCR output as an independent, primary input modality.

In early visual question answering (VQA) frameworks, Amanpreet Singh et al. pioneered this with the LoRRA architecture in Towards VQA Models That Can Read, which extracts scene text strings and maps them into semantic word embeddings alongside standard visual features. More recently, this has evolved into prompt-based text infusion. For instance, in the TRINS framework proposed by Ruiyi Zhang et al. in TRINS: Towards Multimodal Language Models that Can Read, OCR-extracted text strings are directly concatenated with user prompts and visual tokens. By presenting the language model with legible text strings, the system completely bypasses the resolution limits of the visual encoder.

[Raw Image] ----> [Visual Encoder] -------> [Visual Tokens] -----\
                                                                  +---> [Language Model]
[Raw Image] ----> [OCR Engine] ----------> [Explicit Text] -----/

For complex social media media like memes or advertisements, the interaction becomes tri-modal. Models must simultaneously parse the global visual scene, the accompanying caption, and the text physically embedded within the image. As shown by Poorav Desai et al. in Nice Perfume. How Long Did You Marinate in It? Multimodal Sarcasm Explanation and Naoaki Okazaki et al. in SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning, tri-stream encoding architectures process these three distinct inputs through specialized encoders to capture the full semantic interplay.


Mitigating Hallucinations in Technical Domains

The value of explicit text extraction becomes glaringly obvious in technical domains like chart understanding and code generation. When reading a scientific plot, missing a single digit on an axis label completely changes the interpretation of the data.

When models rely solely on raw pixel representations, they often suffer from data hallucinations—confidently misquoting percentages or misreading legends. Xudong Yang et al. demonstrated in AskChart: Universal Chart Understanding through Textual Enhancement that integrating explicit OCR tokens alongside visual features grounds the model's reasoning in precise semantic context, drastically reducing hallucinations in chart question answering. Similarly, in the context of generating code from scientific plots, Chengyue Wu et al. showed in Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots that OCR-augmented visual perception compensates for the inherent resolution limits of standard vision backbones, allowing models to accurately capture dense, small-scale text.


Resolving Semantic Incongruity

Beyond merely reading data, explicit text extraction is critical for high-level pragmatic reasoning, such as detecting sarcasm, hate speech, or misinformation. In these contexts, bad actors or internet culture frequently employ "subversion tactics"—where the text embedded inside a meme or image directly contradicts or alters the meaning of the external caption.

Hongliang Pan et al. addressed this semantic gap in Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection by integrating OCR into multimodal architectures to explicitly model intra- and inter-modality incongruities. Without this explicit extraction, a model might perceive a benign image and a benign caption independently, completely missing the toxic or sarcastic clash hidden within the image's embedded text. This dual-stream alignment has proven vital across a range of safety and verification tasks, from identifying multimodal hate speech (Raúl Gómez et al. in Exploring Hate Speech Detection in Multimodal Publications) to expanding search queries to retrieve fact-checking articles for image-heavy misinformation (Ben Vo et al. in Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News).


The Bidirectional Synergy of Vision and Text

A common critique of modular OCR pipelines is that they are brittle: if the OCR engine misreads a character due to low resolution or unusual fonts, the error propagates downstream. However, recent work demonstrates a powerful bidirectional synergy between the visual and textual modalities.

Instead of treating OCR as a static, one-way preprocessing step, modern frameworks use the visual reasoning capabilities of the LMM itself as a corrective layer. For example, Rawisara Lohanimit et al. developed a multi-stage pipeline in Privacy in Image Datasets: A Case Study on Pregnancy Ultrasounds where a vision-language model corrects character recognition errors by cross-referencing the noisy OCR text with the surrounding visual context of the original image. Similarly, Benjamin Gutteridge et al. showed in Judge a Book by its Cover: Investigating Multi-Modal LLMs for Multi-Page Handwritten Document Transcription that providing an LMM with the OCR text of an entire document alongside a single page's image allows the model to learn the writer's style and dynamically resolve handwriting ambiguities.

Furthermore, text isn't just floating in a vacuum; its position matters. To prevent the loss of spatial context, frameworks like the one proposed by Yuqi Bu et al. in Bridging the Gap between Expression and Scene Text for Referring Expression Comprehension (Student Abstract) explicitly bind detected text strings to specific spatial coordinates within the visual feature map, transforming raw pixels into spatially grounded semantic tokens.

Key Foundational Work

Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrieval (2017) — contributed 2 concept elements to this direction.
Application of Information Retrieval Technologies to Presentation Slides (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ocr_augmented_multimodal_fusion_38a9abe5,
  title = {OCR-augmented multimodal fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ocr-augmented-multimodal-fusion/rdr_38a9abe54620431ab494d5eadeae1ee6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_ocr_augmented_multimodal_fusion_38a9abe5_v1,
  title = {OCR-augmented multimodal fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ocr-augmented-multimodal-fusion/rdr_38a9abe54620431ab494d5eadeae1ee6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake