Traditional Natural Language Processing (NLP) treats documents as linear sequences of tokens. However, real-world documents—invoices, forms, scientific papers, and web pages—are "visually rich." In these formats, the meaning of a word is often defined as much by its physical position and visual styling as by its linguistic context.
Research in this field has shifted from sequential "OCR then NLP" pipelines toward unified multi-modal architectures that treat spatial layout and visual features as first-class citizens alongside text.
The Three Pillars of Document Understanding
To "read" a document like a human, a model must integrate three distinct modalities. Srikar Appalaraju, Bhavan Jasani, and Bhargava Urala Kota formalize this in DocFormer, arguing that the interplay between these signals is essential for interpreting complex structures like forms.
- Textual Semantics: The raw characters and words extracted via Optical Character Recognition (OCR).
- Visual Features: The "look" of the document—fonts, colors, lines, logos, and textures. Hieu M. Vu and Diep Thi-Ngoc Nguyen demonstrate that even without reading text, visual cues like lines and font weights are powerful enough to identify key-value relationships in forms.
- Spatial Layout: The 2D coordinates (bounding boxes) of every element. Ruihua Song and colleagues established early on that a block's spatial position and size are the most significant indicators of its importance on a page.
Spatiality as Semantics
In a structured document, proximity implies a relationship. A label like "Total" is semantically linked to the number physically located to its right or below it, regardless of where those words appear in a flattened text string.
Xiaojing Liu and Feiyu Gao addressed this by representing documents as graphs. In their Graph Convolutional approach, nodes represent text segments and edges encode spatial relationships. This allows the model to "see" the document's geometry, significantly outperforming sequence-only models that lose this structural context. Similarly, Colin Lockard and Hannaneh Hajishirzi use Graph Attention Networks in ZeroShotCeres to extract relations from web pages by treating the DOM tree and visual layout as a multi-modal graph.
From Pipelines to Unified Transformers
The "OCR + NLP" paradigm often fails because errors in the first step (e.g., a typo in OCR) propagate to the second. Minesh Mathew and Dimosthenis Karatzas highlight a major paradigm shift in the DocVQA challenges: the rise of multi-modal transformers.
These models, such as LayoutLM or LongFin, embed text, visual patches, and 2D coordinates into a unified representation space. By training on tasks like "masked visual-language modeling," they learn to reconstruct missing text using both the surrounding words and the visual/spatial context. Panfeng Cao and Zaiqiao Meng take this further with GenKIE, a generative model that can actually correct OCR errors by understanding what a word should be based on its position in a standard form.
The Native Multi-modal Future
Recent work suggests that even the most sophisticated text-based parsing (like converting a PDF to Markdown) might be a bottleneck. Kostadin Cvejoski and Rafet Sifa found that native multi-modal processing—feeding raw images directly into models like Gemini—consistently outperforms pipelines that use intermediate text formats.
This "pixels-to-answers" approach allows the model to leverage the full, unadulterated visual grammar of the document. As Yuxiao Huang demonstrates in routing optimization tasks, "looking" at a layout allows models to capture spatial heuristics and geometric relationships that are nearly impossible to infer from coordinates or text alone.