Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser

Yuta Koreeda, Christopher D. Manning

Abstract

Standard NLP pipelines often struggle with "Visually Structured Documents" (VSDs) like PDFs, where logical hierarchies are conveyed through layout rather than just text. This paper introduces a multimodal transition parser that reconstructs a document's logical tree by fusing visual, textual, and semantic features. The system significantly outperforms standard tools like PDFMiner, achieving a paragraph boundary detection F1 score of 0.953 compared to the baseline's 0.739.

Introduction

While NLP research typically assumes clean, raw text, real-world documents—especially in the legal domain—are often locked in PDFs or complex layouts. Existing preprocessing tools generally fall into two extremes: micro-level word segmentation or macro-level layout analysis (e.g., distinguishing a figure from a text block).

The "missing middle" is fine-grained logical structure analysis. This involves identifying paragraph boundaries and their hierarchical relationships (e.g., knowing that clause 2.1.1 is a child of section 2.1). Without this structure, information extraction systems struggle to associate definitions with the correct clauses or to filter out "debris" like page numbers and headers that interrupt the text flow.

Methodology: Transition Parsing for Document Structure

The authors formulate logical structure analysis as a transition parsing problem. Given a sequence of text blocks, the system predicts a transition label between each consecutive pair. This approach allows the parser to build a hierarchical tree representation of the document.

Overview of the logical structure analysis and its tree-based formulation

The parser predicts one of five transition labels:

  • continuous: Blocks belong to the same paragraph.
  • consecutive: Start of a new paragraph at the same hierarchical level.
  • down: Start of a new paragraph that is a child of the previous one.
  • up: Start of a new paragraph at a higher hierarchical level (requires a pointer to the parent block).
  • omitted: The block is debris (headers, footers, etc.) and should be ignored.

Multimodal Feature Fusion

To make these predictions, the parser uses a Random Forest classifier trained on a multimodal feature set:

  1. Visual Cues: Indentation, line spacing, and alignment. The system uses dynamic margin detection to remain robust across different document formats.
  2. Textual Cues: Numbering patterns (e.g., "1.1" followed by "1.2"), punctuation, and specific keywords like "whereas."
  3. Semantic Cues: Language model coherence (using GPT-2) helps identify debris. If a block is debris, the transition from the previous block to the next one will likely be more semantically coherent than the transition involving the debris block itself.

Experimental Setup and Datasets

The study targets four distinct types of VSDs to demonstrate customizability: English PDF contracts (NDAs), English executive orders, English text-based NDAs from the EDGAR database, and Japanese PDF NDAs. The authors manually annotated 142 documents to create a gold-standard dataset of logical trees.

Examples of the four types of VSDs used in the dataset

Results and Discussion

The multimodal parser consistently outperformed baselines across all metrics. On English PDF contracts, the system achieved a paragraph boundary detection F1 of 0.953, vastly outperforming PDFMiner (0.739) and a rule-based numbering baseline (0.772). Furthermore, the system was significantly better at identifying "Sibling" and "Descendant" relationships, which are critical for understanding legal nesting.

The "omitted" label effectively filtered out headers and footers, improving the input quality for downstream NLP tasks. However, limitations remain; the system occasionally fails on documents where section titles are bolded or underlined but lack indentation, misclassifying them as continuous text.

Conclusion

This work demonstrates that multimodality is essential for document understanding; neither visual nor textual cues are sufficient on their own. By treating document structure as a series of transitions, the authors provide a robust framework for extracting logical hierarchies from complex layouts. Future work may involve fine-tuning language models specifically on legal domains or moving toward end-to-end deep learning approaches.

Summary updated

Abstract

While many NLP pipelines assume raw, clean texts, many texts we encounter in the wild, including a vast majority of legal documents, are not so clean, with many of them being visually structured documents (VSDs) such as PDFs. Conventional preprocessing tools for VSDs mainly focused on word segmentation and coarse layout analysis, whereas fine-grained logical structure analysis (such as identifying paragraph boundaries and their hierarchies) of VSDs is underexplored. To that end, we proposed to formulate the task as prediction of “transition labels” between text fragments that maps the fragments to a tree, and developed a feature-based machine learning system that fuses visual, textual and semantic cues. Our system is easily customizable to different types of VSDs and it significantly outperformed baselines in identifying different structures in VSDs. For example, our system obtained a paragraph boundary detection F1 score of 0.953 which is significantly better than a popular PDF-to-text tool with an F1 score of 0.739.
BibTeX
@inproceedings{koreeda2021capturing,
  author = {Yuta Koreeda and Christopher D. Manning},
  title = {{Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser}},
  booktitle = {NLLP @ EMNLP},
  year = {2021},
}
Figure