Layout Aware Representations for Visually Rich Documents

440 concepts267 papers5 proposals

Standard natural language processing models treat text as a one-dimensional sequence of tokens. While this works well for novels or conversational transcripts, it breaks down when applied to visually rich documents like invoices, academic papers, forms, and web pages. In these formats, meaning is heavily dictated by two-dimensional spatial arrangement.

To bridge this gap, a prominent research direction focuses on layout-aware representation learning. Instead of discarding spatial context during text serialization, these techniques integrate 2D geometric coordinates directly into neural architectures, allowing models to simultaneously "read" and "see" document structure.

The Failure of One-Dimensional Serialization

When a human reads a multi-column journal article or a financial table, they do not read continuously from left to right across the entire page. They use visual cues—such as column boundaries, white space, and vertical alignment—to navigate the text.

Traditional NLP pipelines force these complex 2D structures into a 1D string via Optical Character Recognition (OCR) engines. This serialization process strips away crucial context. For instance, a table cell value might end up thousands of tokens away from its corresponding header in a linearized string, even though they sit directly adjacent to each other on the page.

As noted by Kyle Lo et al. in their work on PaperMage, treating a document as either a flat text string or a raw image creates an artificial divide. Pure text models lose visual context, while pure vision models struggle with fine-grained linguistic structure. In benchmark evaluations of multi-page documents, Jordy Van Landeghem demonstrated in DUDE that text-only large language models consistently fail when navigating non-linear layouts, underscoring the necessity of spatial awareness.

Injecting Geometry into Self-Attention

The most common mechanism for preserving layout is the injection of 2D spatial coordinates into the model's embedding layer. In a standard Transformer, a token's representation is the sum of its semantic word embedding and a 1D positional embedding (which denotes its order in the sequence). Layout-aware models expand this by adding 2D bounding box coordinates.

For every word or text segment, the OCR engine extracts a bounding box defined by its top-left and bottom-right coordinates: $[x_{min}, y_{min}, x_{max}, y_{max}]$. These coordinates are quantized into integers (typically scaled from 0 to 1000) and passed through embedding layers to generate spatial coordinate vectors.

$$\text{Embedding} = \text{Text\_Emb}(t_i) + \text{Pos\_Emb}(i) + \text{X\_Emb}(x_{min}, x_{max}) + \text{Y\_Emb}(y_{min}, y_{max})$$

By adding these spatial vectors, the self-attention mechanism can natively calculate relationships based on physical proximity. When computing the attention score between a query token and a key token, the model evaluates not just whether the words are semantically related, but whether they are physically aligned on the page.

This approach, popularized by architectures like LayoutXLM introduced by Tengchao Lv et al. in the XFUND framework, allows the model to learn universal structural patterns. Similarly, Wonseok Hwang et al. utilized 2D coordinates, character height, and orientation in WYVERN to build an end-to-end information extraction system that bypasses rigid text serialization.

By anchoring features to 2D space rather than a sequential reading order, models become highly robust against OCR reading-order errors. For instance, Mingfei Gao et al. showed in DocQueryNet that pre-training models exclusively on 2D geometric coordinates rather than sequential text streams drastically improves key-value extraction from complex forms. To optimize this process for modern vision-language architectures, Roy Ganz proposed DocVLM, which utilizes a specialized encoder to process text tokens alongside their 2D coordinates, preserving structural context without incurring the high computational cost of processing ultra-high-resolution images.

Discretizing the Page as a Semantic Grid

An alternative paradigm avoids coordinate embeddings altogether by treating the document page as a literal 2D canvas where text tokens are "painted" onto a grid.

In Chargrid, Anoop R. Katti et al. mapped a document page to a two-dimensional grid of character indices. Instead of an RGB image where pixels represent colors, a Chargrid is a matrix where each cell contains an integer corresponding to a specific character at that physical location. This representation allows standard Convolutional Neural Networks (CNNs) to process the document. The CNN uses its natural inductive bias for spatial locality to detect document elements (like total amounts on an invoice) based entirely on their shape and layout configuration, while preserving the exact textual identity.

Standard Image (RGB):      [Pixel] [Pixel] [Pixel]
Linearized Text:           "Total Amount: $150.00"
Chargrid / Semantic Grid:  [ T ] [ o ] [ t ] [ a ] [ l ] [   ] [ $ ] [ 1 ] [ 5 ] [ 0 ]

This concept was extended by Weihong Lin et al. in ViBERTgrid, which maps dense, contextualized word embeddings from a language model onto a 2D grid. This approach fuses rich semantic representations with pixel-level spatial coordinates, allowing computer vision backbones to perform key information extraction directly on the combined representation.

Why Spatial Awareness Unlocks Document Semantics

Layout-aware representations work because they align the model's input space with the human design choices that structure information. This spatial grounding is critical across several domains:

  • Table and Grid Parsing: In a borderless table, a number's meaning is entirely defined by its vertical alignment with a column header and its horizontal alignment with a row header. By explicitly encoding directed spatial relationships, models can immediately resolve these dependencies. For example, Lu Chen et al. developed TIE to capture vertical and horizontal alignments on web pages, allowing the model to accurately parse comparison grids that confuse sequential readers.
  • Key-Value Pairing: In forms, a label (e.g., "Invoice Date") might appear to the left of, or directly above, its corresponding value. Spatial representations allow the model to learn geometric proximity rules that hold true regardless of the specific vocabulary used in the form.
  • Non-traditional Text Formats: Spatial layout dictates rhetorical structure even outside of business documents. In internet memes, Md Shad Akhtar et al. demonstrated that segmenting text based on its physical location (top text vs. bottom text) is essential for capturing the relationship between a setup and a punchline, proving that spatial positioning functions as a primary signal for semantic interpretation.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multimodal_document_representation_b15c2afb,
  title = {multimodal document representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-document-representation/rdr_b15c2afba3d54ecd995399449ba3726b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multimodal_document_representation_b15c2afb_v1,
  title = {multimodal document representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-document-representation/rdr_b15c2afba3d54ecd995399449ba3726b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake