When we read a document, we do not just process a linear sequence of words. We interpret a rich, multi-dimensional canvas of visual signals: the bold weight of a section header, the relative column positions in a financial table, the color-coded highlights of a spreadsheet, and the proximity of a caption to a line chart. This spatial and typographic arrangement forms a "visual grammar" that is essential to understanding the document's underlying logic.
For decades, the standard pipeline for automated document intelligence has ignored this visual grammar. Traditional systems rely on a text-centric paradigm: they run Optical Character Recognition (OCR) or PDF parsers to extract raw text, flatten that text into a one-dimensional string of tokens, and pass those tokens to a language model.
This text-centric approach introduces a severe bottleneck. By stripping away the visual structure, the document is fundamentally degraded before the model ever encounters it.
The Lossy Flattening of Structured Documents
The core flaw of traditional text extraction is that it treats multi-dimensional layouts as flat books. As Le Qi et al. demonstrate in their work on open-domain document visual question answering (DuReadervis), converting structured documents into raw text strings discards critical visual cues like font size, spatial positioning, and table structures. This makes it incredibly difficult for models to reconstruct the hierarchical importance and contextual relationships inherent in complex layouts.
Consider how information is encoded in common document types:
- Spreadsheets and Tables: In semi-structured data, visual styling often is the data. Markus Schröder et al. (Spread2RML) point out that critical semantic information is frequently encoded through visual styling—such as cell colors or font weights—rather than explicit text. Traditional parsers ignore these cues entirely.
- Charts and Infographics: Reverse-engineering a chart by simply extracting its raw data table misses the point. Zhiguang Zhou et al. (ChartKG) note that ignoring visual encodings like color meanings or spatial relationships results in a total loss of the designer's intended context and high-level messaging.
- Web Pages and Financial Comments: Treating web content as flat text ignores the HTML DOM tree and rendering coordinates. Qifan Wang et al. (WebFormer) show that lacking access to visual rendering information like font size and absolute screen coordinates leads to errors in identifying element hierarchies. Similarly, Huadai Liu et al. (AntCritic) emphasize that text-based analysis misses the logical emphasis provided by HTML styling in free-form financial comments.
Furthermore, this text-centric blindness creates severe security and reliability vulnerabilities. Because automated text parsers extract text regardless of its visual rendering, they are entirely blind to formatting cues like text color or transparency. Zekai Huang et al. (Too Easily Fooled?) exploit this exact flaw, showing that prompt injections written in invisible white text within PDFs can easily manipulate Large Language Models (LLMs) without human detection.
Traditional Pipeline (Lossy):
[Document Image] ──> [OCR / Text Extraction] ──> [Flat Text String] ──> [LLM]
(Destroys layout, fonts, colors, and charts)
Vision-Based Pipeline (Preserved):
[Document Image] ─────────────────────────────────────────────────────> [VLM / Pixel Model]
(Retains 100% of visual-structural context)
Shifting from Tokens to Pixels
To overcome this bottleneck, a new paradigm has emerged: processing documents directly as images. By shifting the input space from text tokens to raw pixels, models can preserve 100% of the document's visual and structural fidelity.
Early foundations for this shift were established by Adam W Harley et al. (Evaluation of Deep Convolutional Nets...), who proved that deep convolutional features learned from natural images transfer remarkably well to black-and-white text layouts, outperforming hand-crafted text features. More recently, self-supervised frameworks like the Document Image Transformer (DiT) introduced by Junlong Li et al. (DiT) leverage domain-specific image tokenizers to preserve the sharp edges and text structures essential for precise layout analysis.
This pixel-first philosophy has culminated in fully autoregressive language models that bypass text vocabularies entirely. For instance, Yekun Chai et al. (PixelGPT) train models to predict the next patch of a rendered document image rather than the next subword token.
By operating directly on pixels, these models bypass the vocabulary bottleneck and naturally capture layout, font styles, and color-coded highlights. For historical or poorly preserved documents where OCR fails entirely, this approach is transformative. Nadav Borenstein (PHD) demonstrates that learning to reconstruct masked pixel patches rather than text tokens retains vital typographical and layout context that traditional text-first approaches discard.
Visual Retrieval-Augmented Generation
The most immediate and impactful application of this pixel-first paradigm is in Retrieval-Augmented Generation (RAG). Traditional RAG pipelines index text chunks; Visual RAG (Vi-RAG) pipelines index document screenshots or page images.
Zheng Liu et al. (Any Information Is Just Worth One Single Screenshot) formalize this concept under Visualized Information Retrieval (Vis-IR), treating screenshots as a universal format for multimodal search. By embedding the entire visual page, systems avoid the lossy text-parsing stage entirely.
As Shi Yu et al. (VisRAG) and Ryota Tanaka et al. (VDocRAG) demonstrate, using Large Vision-Language Models (LVLMs) to embed and generate directly from document images yields massive improvements over text-based systems when dealing with complex PDFs, charts, and multi-column reports.
Why Visual Retrieval Works: Late Interaction on Patches
The breakthrough that makes visual document retrieval computationally efficient in production is the application of late interaction mechanisms to image patches.
In a standard bi-encoder model, an entire document page is compressed into a single vector embedding. This compression is incredibly lossy for a dense, visually rich PDF page. Models like ColPali, introduced by Manuel Faysse et al. (ColPali), solve this by feeding the document page through a vision transformer to obtain a multi-vector representation—essentially an embedding for every small patch of the image.
During retrieval, a text query is not compared against a single page vector. Instead, a late interaction operator (like MaxSim) aligns each token in the text query with the most relevant visual patches on the page.
Query: "What was the revenue in Q3?"
│
├──► "revenue" ──────┐
└──► "Q3" ──────┐ │
▼ ▼
┌───┬───┬───┬───┐
│ │ │ │ │
├───┼───┼───┼───┤
│ │ █ │ █ │ │ ◄── High-similarity visual patches
├───┼───┼───┼───┤ containing the Q3 Financial Table
│ │ │ │ │
└───┴───┴───┴───┘
Document Page Patches
This architecture allows the model to perform fine-grained matching between textual concepts and visual regions without needing an explicit text transcription. Gabriel de Souza P. Moreira et al. (Nemotron ColEmbed V2) have scaled this approach, utilizing bidirectional attention and hard-negative mining to optimize these multi-vector models for production-scale visual document retrieval.
To mitigate the storage and compute costs of maintaining millions of patch vectors, frameworks like the one proposed by Hao Sun et al. (Unveil) use knowledge distillation. They train a powerful joint visual-textual model and distill its capabilities into an efficient, vision-only student model, allowing systems to completely skip costly OCR at inference time while maintaining high accuracy.
Grounding and Verification
Beyond raw performance metrics, the shift to vision-based document processing fundamentally changes how AI systems interact with human experts.
In specialized fields like history, law, or medicine, a transcribed text string is often insufficient because it lacks authenticity and context. Minesh Mathew et al. (Asking questions on handwritten document collections) argue that the original visual appearance of a manuscript provides necessary context and verification grounding that a simple transcription lacks. Similarly, Yi Du et al. (Autodive) find that allowing domain experts to annotate directly on the visual layout of scientific PDFs drastically reduces cognitive load and labeling errors compared to working with plain text.
Visual RAG models exploit this visual grounding to provide explicit source attribution. For example, the VISA framework introduced by Xueguang Ma et al. (VISA) uses a VLM to generate an answer and simultaneously output bounding box coordinates that highlight the exact evidence on the document screenshot.
Generated Answer: "The net profit increased by 12% in 2024."
Source Attribution: [Page 4, Bounding Box: (x1, y1, x2, y2)] ──► Draws red box directly
on the table image
This visual grounding transforms the user experience. Instead of forcing a human to read through pages of text to verify an LLM's claim, the system points directly to the visual artifact—the exact row of a table or segment of a chart—making verification instantaneous.
By treating documents as images and leveraging the spatial reasoning capabilities of modern Vision-Language Models, the field is successfully moving past the text-centric extraction bottleneck. The result is a generation of document intelligence tools that see the page exactly as a human reader does, preserving the rich visual grammar that makes complex documents understandable.
Key Foundational Work
Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval (2015) — contributed 1 concept elements to this direction.
Enhancing document structure analysis using visual analytics (2010) — contributed 1 concept elements to this direction.