XDoc: Unified Pre-training for Cross-Format Document Understanding

Jingye Chen, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

Introduction

Document understanding is a cornerstone of modern Natural Language Processing (NLP), yet the field remains highly fragmented. Currently, practitioners must deploy specialized models for different document formats: RoBERTa for plain text, LayoutLM for scanned documents, and MarkupLM for web pages. This fragmentation leads to two primary issues: knowledge silos, where models cannot share conceptual similarities across formats, and resource inefficiency, as maintaining multiple large-scale models is computationally expensive.

XDoc addresses these challenges by introducing a unified pre-training framework capable of mastering plain text, document images (PDFs/scans), and web pages within a single architecture. By leveraging a shared Transformer backbone and lightweight adaptive layers, XDoc achieves state-of-the-art performance while significantly reducing the total parameter footprint.

Unified Architecture

The core innovation of XDoc is its ability to process diverse formats through a single, shared backbone. The authors observed that in specialized models, over 90% of the parameters reside in the word embedding and Transformer layers. XDoc capitalizes on this by sharing these components across all formats, facilitating the learning of universal language representations.

The XDoc unified architecture for plain text, document text, and web text.

Format-Specific Inputs and Adaptive Layers

To accommodate the unique structural characteristics of each format, XDoc utilizes specialized embedding heads:

  • Plain Text: Employs standard 1D position embeddings.
  • Document Text: Integrates 2D coordinates (bounding boxes) to capture visual layout.
  • Web Text: Uses XPath strings to represent the hierarchical DOM structure of HTML.
Examples of input representations for plain text, document text, and web pages.

Rather than simply summing these embeddings, XDoc introduces Adaptive Layers—lightweight Linear-ReLU-Linear sequences. These layers act as specialized translators, processing format-specific information before it enters the shared Transformer backbone. This approach provides the necessary distinction between formats while maintaining a universal backbone, adding only 2.7% to the total parameter count.

Experimental Setup

XDoc was pre-trained using Masked Language Modeling (MLM) on a massive, balanced corpus:

  • Plain Text: 213,713 files from Wikipedia, BookCorpus, and OpenWebText.
  • Document Text: 11 million scanned pages from the IIT-CDIP collection.
  • Web Text: 24 million web pages from Common Crawl.

The model was evaluated on industry-standard benchmarks, including GLUE and SQuAD for plain text, FUNSD and DocVQA for document understanding, and WebSRC for web-based tasks.

Key Results and Insights

XDoc demonstrates that a "jack-of-all-trades" model can indeed be a master of all. Training on a mixture of documents and web pages not only maintains but often improves performance on plain text tasks.

  • Parameter Efficiency: A single XDoc model (146M parameters) replaces three specialized models totaling 398M parameters—a 63.3% reduction in total footprint.
  • Superior Performance: XDoc achieved an 89.4 F1 score on the FUNSD task, significantly outperforming the specialized LayoutLM baseline (79.3). It also reached 74.8 F1 on WebSRC and 72.7 ANLS on DocVQA.
  • Balanced Training: Empirical results suggest that a 1:1:1 balanced ratio of the three document types during pre-training is critical for optimal average performance across all downstream tasks.

Conclusion and Future Work

XDoc represents a significant step toward universal document understanding. While highly effective, the current model relies on metadata (coordinates/XPaths) rather than raw pixels. Future work could involve incorporating image features to bridge the "vision gap" or exploring more complex structural pre-training tasks. Additionally, expanding the framework to include formats like Microsoft Word or PowerPoint remains an open area for research.

Summary updated

Abstract

The surge of pre-training has witnessed the rapid development of document understanding recently. Pre-training and fine-tuning framework has been effectively used to tackle texts in various formats, including plain texts, document texts, and web texts. Despite achieving promising performance, existing pre-trained models usually target one specific document format at one time, making it difficult to combine knowledge from multiple document formats. To address this, we propose XDoc, a unified pre-trained model which deals with different document formats in a single model. For parameter efficiency, we share backbone parameters for different formats such as the word embedding layer and the Transformer layers. Meanwhile, we introduce adaptive layers with lightweight parameters to enhance the distinction across different formats. Experimental results have demonstrated that with only 36.7% parameters, XDoc achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models, which is cost effective for real-world deployment. The code and pre-trained models will be publicly available at \url{https://aka.ms/xdoc}.
BibTeX
@inproceedings{chen2022xdoc,
  author = {Jingye Chen and Tengchao Lv and Lei Cui and Cha Zhang and Furu Wei},
  title = {{XDoc: Unified Pre-training for Cross-Format Document Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=W10zSjrWt8o},
}
Figure