Enhancing Visually-Rich Document Understanding via Layout Structure Modeling

Qiwei Li, Zuchao Li, Xiantao Cai, Bo Du, Hai Zhao

Introduction

Visually-Rich Document Understanding (VRDU) is the task of interpreting scanned forms, receipts, and reports. While modern models like the LayoutLM family have successfully combined text and visual cues, they often struggle with complex layouts where physical proximity does not align with logical relationships.

A complex trauma registry form illustrating the challenges of mapping spatial coordinates to logical document structure.

Existing models suffer from two major "blind spots":

  1. Reading Order Mismatch: Standard top-to-bottom, left-to-right scanning fails for multi-column forms or non-linear layouts.
  2. Spatial vs. Structural Confusion: 2D coordinates are often a poor proxy for logical hierarchy. A header may be physically close to an unrelated text box but logically connected to a paragraph much further down.

Methodology: GraphLayoutLM

GraphLayoutLM addresses these issues by injecting explicit structural knowledge into a Transformer-based architecture. The model moves beyond simple coordinates to model the document's actual structural hierarchy.

The GraphLayoutLM framework, showing the transition from a document to a layout graph, followed by graph reordering and masking within a multi-modal transformer.

The framework introduces three key strategies:

  • Layout Graph Modeling: The document is treated as a hierarchical tree. Text lines are grouped into paragraphs and sections, establishing "Parent-Child" and "Sibling" relationships.
  • Graph Reordering Strategy: Instead of using the raw OCR sequence, the model uses a Depth-First Search (DFS) traversal of the layout tree. This ensures that logically related text nodes (like a question and its corresponding answer) are placed closer together in the input sequence.
  • Graph Masking Strategy: A Graph-aware Multi-head Self-Attention layer uses an adjacency matrix derived from the layout graph to bias the attention mechanism. This forces the model to focus on structurally related nodes and ignore irrelevant noise from distant, unrelated text.

Key Results and Insights

GraphLayoutLM achieved new State-of-the-Art (SOTA) performance across several standard VRDU benchmarks:

  • FUNSD: Achieved an F1 score of 94.39 (Large), significantly outperforming LayoutLMv3.
  • CORD: Reached an F1 of 97.75, setting a new benchmark for receipt key information extraction.
  • XFUND (Chinese): Achieved an F1 of 93.56, proving the structural approach works across different languages.

The core insight of this research is that structure is more important than coordinates. By explicitly modeling the hierarchy, the model doesn't have to "guess" logical connections. Furthermore, the reordering strategy proves that the sequence in which a model "reads" a document significantly impacts its comprehension.

Limitations and Future Work

The model currently depends heavily on the quality of initial OCR and layout analysis tools to construct the layout graph. Future work will explore more advanced, automated document hierarchy recognition to integrate graph construction directly into the end-to-end learning process.

Summary updated

BibTeX
@inproceedings{li2023enhancing,
  author = {Qiwei Li and Zuchao Li and Xiantao Cai and Bo Du and Hai Zhao},
  title = {{Enhancing Visually-Rich Document Understanding via Layout Structure Modeling}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612327},
}
Figure