Visual Detection with Context for Document Layout Analysis

Carlos Soto, Shinjae YooEMNLP 2019 · OpenReview

Abstract: Carlos Soto, Shinjae Yoo. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction: The PDF Extraction Problem

Extracting structured data from scientific publications is a persistent challenge in data science. Traditional PDF-to-text tools often struggle with multi-column layouts, resulting in jumbled text blocks, out-of-order sentences, and misidentified headers. These tools typically rely on messy PDF metadata or brittle, rule-based heuristics to distinguish between body text, figure captions, and titles.

To address these issues, researchers at Brookhaven National Laboratory have proposed a purely visual approach to Document Layout Analysis (DLA). By treating document regions—such as abstracts, figures, and tables—as "objects" to be detected within an image, the system becomes more robust across various languages and formatting styles, avoiding the errors introduced by poor OCR or corrupted metadata.

Methodology: A Vision-Based Approach

The core of this approach is the adaptation of Faster R-CNN, a standard two-stage object detection framework, to the domain of document images. The process begins by rendering PDF pages as JPEG images, which are then processed through a feature-extraction network.

The proposed model architecture, integrating contextual features into a Faster R-CNN framework.

The framework consists of two primary stages:

  1. Region Proposal Network (RPN): Scans the page image to identify candidate bounding boxes likely to contain content.
  2. Classification Network: Identifies the specific class of content within those boxes.

The critical innovation in this work is the inclusion of contextual features. While standard object detectors rely solely on pixel data within a bounding box, the authors recognized that spatial metadata is vital for DLA. For instance, an abstract and a body paragraph may look visually identical, but their position on the page is a strong indicator of their identity. The model appends three contextual data points to the feature vector: the relative page number, the X and Y coordinates of the box, and the box's dimensions (width and height).

The Dataset

Finding that existing datasets like GROTOAP2 were too granular or noisy for high-level layout analysis, the authors developed a new benchmark dataset. They manually annotated 100 scientific articles (totaling 822 pages) across nine distinct classes: Title, Authors, Abstract, Body, Figure, Figure Caption, Table, Table Caption, and References. This dataset provides a clean, consistent foundation for training and evaluating visual DLA models.

Results: Context is King

The addition of contextual features yielded a dramatic improvement in detection accuracy. The Mean Average Precision (mAP) jumped from 46.4% to 70.3%, representing a 51.6% relative improvement over the baseline Faster R-CNN model.

Performance metrics showing the impact of contextual features on Mean Average Precision (left) and per-class precision (right).

Key performance highlights include:

  • Superior Accuracy: The model outperformed other state-of-the-art detectors, including YOLOv3 (68.9% mAP) and RetinaNet (58.1% mAP).
  • High Precision in Core Areas: The system achieved nearly perfect results for "Body" text, with a precision of 93.58%.
  • Efficiency: The visual approach is significantly faster than traditional methods. Processing a single article takes approximately 0.65 seconds, making it 14 times faster than CERMINE, a popular text-based extraction tool.

Limitations and Future Work

Despite its overall success, the model faces challenges with "small objects." Specifically, it struggled with the Authors class (1.22% precision) and Table Captions (30.8%), likely due to their small physical size and inconsistent placement across different journals.

Future research will focus on expanding the dataset to 1,000 articles, incorporating more diverse sources like arXiv pre-prints. The authors also plan to explore more advanced contextual features, such as "all-region positional information," allowing the model to consider the relative positions of all predicted boxes on a page to better understand the document's overall flow.

Summary updated

BibTeX
@inproceedings{soto2019visual,
  author = {Carlos Soto and Shinjae Yoo},
  title = {{Visual Detection with Context for Document Layout Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=AzBjV5hcRyI},
}
Figure