Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser

Yuta Koreeda, Christopher D. Manning

Abstract

While many NLP pipelines assume raw, clean texts, many texts we encounter in the wild, including a vast majority of legal documents, are not so clean, with many of them being visually structured documents (VSDs) such as PDFs. Conventional preprocessing tools for VSDs mainly focused on word segmentation and coarse layout analysis, whereas fine-grained logical structure analysis (such as identifying paragraph boundaries and their hierarchies) of VSDs is underexplored. To that end, we proposed to formulate the task as prediction of "transition labels" between text fragments that maps the fragments to a tree, and developed a feature-based machine learning system that fuses visual, textual and semantic cues. Our system is easily customizable to different types of VSDs and it significantly outperformed baselines in identifying different structures in VSDs. For example, our system obtained a paragraph boundary detection F1 score of 0.953 which is significantly better than a popular PDF-to-text tool with an F1 score of 0.739.

Introduction

The vast majority of real-world documents, particularly in the legal and administrative domains, are Visually Structured Documents (VSDs). Unlike plain text, the meaning in VSDs is often inextricably linked to visual layout elements such as indentation, numbering, and spatial positioning. While existing layout analysis tools effectively identify text blocks, they often fail to capture the "logical structure"—the hierarchical relationships between these blocks. This failure leads to "debris" (e.g., headers, footers, page numbers) contaminating the text and disrupts the contextual hierarchy necessary for downstream Information Extraction (IE).

Methodology: Multimodal Transition Parsing

We approach logical structure analysis as a transition parsing problem. Rather than simple classification, our system predicts a transition label between consecutive text fragments (b_i and b_{i+1}) to reconstruct a document tree.

The transition parsing framework maps text blocks to a logical tree by predicting labels like 'down', 'continuous', and 'omitted'.

Transition Labels

The parser utilizes five primary labels to define relationships between blocks:

  • Continuous: The blocks belong to the same paragraph.
  • Consecutive: b_{i+1} starts a new paragraph at the same hierarchical level as b_i.
  • Down: b_{i+1} is a child (sub-section) of b_i.
  • Up: b_{i+1} returns to a higher level in the hierarchy, requiring a pointer to the target parent.
  • Omitted: The block is identified as "debris" (e.g., a page number) and is excluded from the logical flow.

Feature Fusion

Our system employs a Random Forest classifier utilizing a multimodal feature set to handle the scarcity of proprietary legal data:

  1. Visual Features: Indentation, line spacing, alignment, and page position.
  2. Textual Features: Punctuation patterns and numbering transitions (e.g., recognizing that "ii)" follows "i)").
  3. Semantic Features: Language model coherence scores derived from GPT-2 to assess the natural flow between blocks and identify jarring transitions caused by debris.

Experimental Setup

To evaluate the system, we introduced a benchmark dataset covering four distinct VSD categories: English PDF contracts (NDAs), English Executive Orders, English plain text contracts (from EDGAR), and Japanese PDF contracts. This diversity tests the system's robustness across different languages and formatting conventions.

Sample documents from the four categories: English PDF contracts, English Executive Orders, English plain text contracts, and Japanese PDF contracts.

Results and Discussion

The proposed multimodal parser significantly outperformed standard baselines across all document types. In paragraph boundary detection on English PDF contracts, the system achieved an F1 score of 0.953, markedly higher than the 0.739 achieved by the popular open-source tool PDFMiner. Furthermore, the system demonstrated robust hierarchical accuracy, achieving a Micro-F1 of 0.784 in identifying ancestor-descendant relationships, compared to 0.440 for a purely visual baseline.

The inclusion of the "omitted" label effectively filters document debris, while the multimodal approach ensures robustness. Future work will explore incorporating typographic features like font weight and expanding the dataset to more diverse domains such as scientific literature and financial reports.

Summary updated

Abstract

While many NLP pipelines assume raw, clean texts, many texts we encounter in the wild, including a vast majority of legal documents, are not so clean, with many of them being visually structured documents (VSDs) such as PDFs. Conventional preprocessing tools for VSDs mainly focused on word segmentation and coarse layout analysis, whereas fine-grained logical structure analysis (such as identifying paragraph boundaries and their hierarchies) of VSDs is underexplored. To that end, we proposed to formulate the task as prediction of "transition labels" between text fragments that maps the fragments to a tree, and developed a feature-based machine learning system that fuses visual, textual and semantic cues.Our system is easily customizable to different types of VSDs and it significantly outperformed baselines in identifying different structures in VSDs. For example, our system obtained a paragraph boundary detection F1 score of 0.953 which is significantly better than a popular PDF-to-text tool with an F1 score of 0.739.
BibTeX
@inproceedings{koreeda2021capturing,
  author = {Yuta Koreeda and Christopher D. Manning},
  title = {{Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser}},
  booktitle = {NLLP @ EMNLP},
  year = {2021},
}
Figure