LayerDoc: Layer-wise Extraction of Spatial Hierarchical Structure in Visually-Rich Documents

Puneet Mathur, Rajiv Jain, Ashutosh Mehra, Jiuxiang Gu, Franck Dernoncourt, Anandhavelu Natarajan, Quan Hung Tran, Verena Kaynig-Fittkau, Ani Nenkova, Dinesh Manocha, Vlad I. Morariu

Introduction: The Challenge of Document Hierarchy

When humans view a complex form or a mobile application interface, we intuitively group elements. We recognize that a label belongs to a specific checkbox, and together they form a logical "Choice Group." However, digital representations like PDFs or scanned images often discard this structural logic, leaving only raw pixels or a flat stream of text characters.

Existing models often treat documents as simple sequences of words. This approach faces a "chicken-and-egg" dilemma: accurate document understanding often requires a correct reading order, yet determining that order requires an understanding of the document's hierarchical structure. Standard "top-to-bottom, left-to-right" heuristics frequently fail in multi-column or nested layouts.

Figure 1: Comparison of standard reading order versus the proposed spatial hierarchy and word grouping approach.

The LayerDoc Framework

To address these issues, the authors propose LayerDoc, a framework that reconstructs document structure through recursive, bottom-up grouping. Instead of predicting the entire hierarchy simultaneously, LayerDoc operates layer-by-layer.

Multimodal Contextual Encoding

The core of LayerDoc is a multimodal encoder that evaluates potential "parent-child" relationships between document elements using three primary signals:

  1. Semantic Cues: Textual content is processed using SentenceBERT (SBERT) to capture the meaning of labels and values.
  2. Spatial Cues: The model utilizes normalized coordinates of bounding boxes to understand geometric proximity.
  3. Visual Cues: Features are extracted directly from the document image via a visual encoder to identify visual boundaries and styles.
Figure 2: The LayerDoc architecture, featuring the multimodal contextual encoder and the link prediction mechanism for hierarchical reconstruction.

Recursive Grouping and Constraint Inference

The process begins with elementary tokens (words or icons). LayerDoc generates candidate parent boxes and uses a Transformer-based architecture to predict the likelihood of a link and classify the parent element type (e.g., "Choice Field" or "Table Row").

Because many overlapping candidate boxes may be generated, the model employs a global optimization step—a variation of the Maximum Independent Set problem—to select the most likely set of non-overlapping parents. These new parent boxes then serve as the "children" for the subsequent layer, repeating until the document is unified under a single root node.

Key Insights

  • Hierarchy Informs Reading Order: By traversing the extracted tree structure, LayerDoc generates a reading sequence that is significantly more accurate than traditional OCR heuristics.
  • Semantic Importance: Integrating SBERT for semantic analysis provided a substantial performance boost (8-14%), confirming that understanding text content is vital for logical grouping.
  • Computational Efficiency: The layer-wise approach is highly efficient, requiring approximately 10x fewer forward passes than models that attempt exhaustive pairwise comparisons across the entire document.

Experimental Results

LayerDoc was evaluated on the Hierarchical Forms, RICO (mobile app layouts), and FUNSD datasets. Key findings include:

  • Accuracy: A 10-15% relative improvement in spatial region classification and group identification over baselines like DocStruct and Form2Seq.
  • Reading Order: An 8-12% improvement in inferred token reading order and semantic word grouping.
  • Robustness: Higher Mean Average Precision (mAP) in reconstructing full document trees, particularly in complex, nested scenarios.

Limitations and Future Directions

While LayerDoc represents a significant advancement, it is susceptible to recursive error propagation, where a mistake in a lower layer (e.g., grouping words into lines) can affect all subsequent higher-level groupings. Future work may explore more sophisticated global optimization or "backtracking" mechanisms to allow the model to revise lower-layer decisions based on higher-level context. Additionally, testing cross-dataset generalization—such as applying models trained on forms to scientific papers—remains an area for further exploration.

Summary updated

Abstract

Digital documents often contain images and scanned text. Parsing such visually-rich documents is a core task for work-flow automation, but it remains challenging since most documents do not encode explicit layout information, e.g., how characters and words are grouped into boxes and ordered into larger semantic entities. Current state-of-the-art layout extraction methods are challenged by such documents as they rely on word sequences to have correct reading order and do not exploit their hierarchical structure. We propose LayerDoc, an approach that uses visual features, textual semantics, and spatial coordinates along with constraint inference to extract the hierarchical layout structure of documents in a bottom-up layer-wise fashion. LayerDoc recursively groups smaller regions into larger semantic elements in 2D to infer complex nested hierarchies. Experiments show that our approach outperforms competitive baselines by 10-15% on three diverse datasets of forms and mobile app screen layouts for the tasks of spatial region classification, higher-order group identification, layout hierarchy extraction, reading order detection, and word grouping.
BibTeX
@inproceedings{mathur2023layerdoc,
  author = {Puneet Mathur and Rajiv Jain and Ashutosh Mehra and Jiuxiang Gu and Franck Dernoncourt and Anandhavelu Natarajan and Quan Hung Tran and Verena Kaynig-Fittkau and Ani Nenkova and Dinesh Manocha and Vlad I. Morariu},
  title = {{LayerDoc: Layer-wise Extraction of Spatial Hierarchical Structure in Visually-Rich Documents}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00360},
}
Figure