Abstract: Documents often contain complex physical structures, which make the Document Layout Analysis (DLA) task challenging. As a pre-processing step for content extraction, DLA has the potential to capture rich information in historical or scientific documents on a large scale. Although many deep-learning-based methods from computer vision have already achieved excellent performance in detecting Figure from documents, they are still unsatisfactory in recognizing the List, Table, Text and Title category blocks in DLA. This paper proposes a VTLayout model fusing the documents' deep visual, shallow visual, and text features to localize and identify different category blocks. The model mainly includes two stages, and the three feature extractors are built in the second stage. In the first stage, the Cascade Mask R-CNN model is applied directly to localize all category blocks of the documents. In the second stage, the deep visual, shallow visual, and text features are extracted for fusion to identify the category blocks of documents. As a result, we strengthen the classification power of different category blocks based on the existing localization technique. The experimental results show that the identification capability of the VTLayout is superior to the most advanced method of DLA based on the PubLayNet dataset, and the F1 score is as high as 0.9599.
Introduction
Document Layout Analysis (DLA) is the critical first step in automated document understanding, involving the identification and categorization of regions such as titles, tables, and paragraphs. While modern computer vision models excel at identifying visually distinct elements like "Figures," they often struggle with text-heavy blocks. To a standard neural network, a List, a Table, and a Paragraph can look remarkably similar—all appearing as rectangular blocks of black text on a white background. "Titles" are particularly difficult because their sparse word count provides little data for purely visual models. VTLayout addresses this by moving beyond purely visual features, incorporating shallow pixel statistics and textual semantics into the classification process.
The VTLayout Framework
The authors propose a two-stage framework that decouples the localization of document elements from their final classification. This allows for a specialized, multi-modal fusion model to handle the nuances of document structure.
Stage 1: Category Block Localization
The model utilizes a Cascade Mask R-CNN to detect the bounding boxes of all potential blocks within a document. This stage focuses exclusively on spatial localization, ensuring high recall for all document elements regardless of their category.
Stage 2: Category Block Classification
Once blocks are localized, they are processed by three specialized feature extractors:
- Deep Visual Feature Extractor (DVFE): Employs a lightweight MobileNetV2 backbone with a Squeeze-and-Excitation (SE) network to capture high-level visual patterns.
- Shallow Visual Feature Extractor (SVFE): A novel component that calculates statistical pixel histograms for each block. This captures intuitive features like the boldness of a Title or the specific whitespace density of a Table.
- Text Feature Extractor (TFE): Uses PaddleOCR to extract text, followed by TF-IDF (Term Frequency-Inverse Document Frequency) weighting. This provides a semantic "sanity check," helping distinguish a "References" section from standard body text.
These three feature vectors are concatenated and processed by a four-layer fully connected network (FCU) to produce the final classification via a Softmax layer.
Key Insights: Why Multi-Modal Fusion Works
A core finding of this research is that deep learning often discards raw pixel data that is actually highly informative for document structures. By re-introducing "shallow" features, the model gains a significant advantage in identifying categories that look visually similar at a high level of abstraction.
As shown in the pixel distribution analysis, categories like Titles often exhibit darker/bolder pixel signatures, while Tables have unique whitespace patterns. By combining these shallow cues with deep visual patterns and semantic text data, VTLayout achieves a more holistic understanding of the document.
Experimental Results
VTLayout was evaluated on PubLayNet, the largest dataset for DLA, containing over 360,000 document images. The model achieved state-of-the-art performance with an F1 score of 0.9599. Notably, the fusion approach significantly boosted performance in historically difficult categories:
- Lists: F1 score reached 0.9177.
- Titles: F1 score reached 0.9411.
Ablation studies confirmed that every component of the fusion—DVFE, SVFE, and TFE—contributed to the final accuracy. Removing any single feature extractor resulted in a measurable drop in performance, proving that multi-modal data is essential for complex document understanding.
Conclusion and Future Work
VTLayout demonstrates that fusing visual and textual features provides a robust solution for the challenges of Document Layout Analysis. While the model sets a new benchmark, challenges remain in handling low-resolution images where OCR quality degrades, and in distinguishing highly complex nested lists. Future research will explore Transformer-based backbones and more efficient, end-to-end architectures for real-time document processing.
Summary updated