XDoc: Unified Pre-training for Cross-Format Document Understanding

Jingye Chen, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

Abstract

The rapid development of document understanding has been driven by the pre-training and fine-tuning paradigm. However, existing models typically target a single document format—such as plain text, visually rich documents (PDFs/scans), or web pages—at a time. This fragmentation leads to knowledge isolation and significant resource redundancy. We propose XDoc, a unified pre-trained model designed to process multiple document formats within a single architecture. By sharing backbone parameters and utilizing lightweight adaptive layers, XDoc achieves state-of-the-art performance across diverse tasks. Remarkably, XDoc requires only 36.7% of the parameters compared to maintaining individual specialized models, offering a cost-effective solution for real-world deployment.

Introduction

In the current natural language processing (NLP) landscape, document understanding is often treated as a series of isolated problems. Processing a novel typically requires a model like RoBERTa (optimized for plain text); extracting data from an invoice requires LayoutLM (optimized for 2D document layouts); and scraping a website requires MarkupLM (optimized for HTML structures).

This "siloed" approach presents two major drawbacks. First, it causes knowledge isolation, where models cannot leverage structural "common sense" across formats (e.g., the relationship between a header and a paragraph is similar in both PDFs and HTML). Second, it creates resource inefficiency. Deploying three separate 100M+ parameter models is computationally expensive and redundant, given that these models share nearly identical underlying architectures, such as word embeddings and Transformer blocks.

The XDoc Architecture

The core innovation of XDoc is its Unified Architecture with Adaptive Layers, which allows a single Transformer backbone to handle three distinct document formats simultaneously.

Shared Backbone

To maximize parameter efficiency, XDoc shares its most computationally expensive components across all formats. This includes the word embedding layer, 1D position embeddings, and the entirety of the Transformer layers. This design is based on the observation that the format-specific components in specialized models typically account for only 2% to 8% of the total parameters.

Format-Specific Embeddings

To accommodate the unique structural cues of different formats, XDoc utilizes specialized input heads:

  • Plain Text: Employs standard 1D sequence embeddings.
  • Document Text: Incorporates 2D position embeddings that represent the spatial coordinates of text boxes on a page.
  • Web Text: Incorporates XPath embeddings to capture the hierarchical location of elements within an HTML tree.

Adaptive Layers

To prevent the shared backbone from becoming "confused" by these varying input types, the authors introduced lightweight Adaptive Layers. These layers consist of a simple Linear-ReLU-Linear sequence that acts as a format-specific tuner. They enhance the distinction of prior knowledge (like 2D positions or XPaths) before the data enters the shared Transformer, adding only negligible latency (approx. 0.8ms) to inference.

Pre-training and Datasets

XDoc was pre-trained using the Masked Language Modeling (MLM) objective on a massive, multi-format corpus:

  • Plain Text: 213,713 files from BookCorpus, Wikipedia, CC-News, and OpenWebText.
  • Document Text: 11 million scanned document pages from the IIT-CDIP collection.
  • Web Text: 24 million web pages derived from Common Crawl.

Research indicated that a balanced "diet" of these formats (a 1:1:1 ratio) provided the best results, preventing the model from developing a bias toward any single document type. Furthermore, initializing XDoc with RoBERTa weights proved more effective than training from scratch, highlighting the importance of a strong linguistic foundation.

Key Results and Performance

XDoc demonstrates that unification does not require a compromise in performance. In fact, cross-format training often yields a synergy that improves results beyond what specialized models achieve.

  • Parameter Efficiency: XDoc (146M parameters) replaces three separate models that would otherwise total 398M parameters—a 63.3% reduction in total footprint.
  • Document Understanding: On the FUNSD benchmark, XDoc achieved an F1 score of 89.4, significantly outperforming the individual LayoutLM baseline of 79.3.
  • Web Extraction: On the WebSRC benchmark, XDoc reached an F1 of 74.8, surpassing the specialized MarkupLM (74.5).
  • General Language Tasks: XDoc maintained competitive performance on standard GLUE and SQuAD benchmarks, proving that adding structural understanding does not degrade its ability to process plain text.

Conclusion and Future Work

XDoc represents a significant step toward a universal document understanding agent. By consolidating multiple formats into a single model, it reduces the barriers to deploying sophisticated NLP tools in resource-constrained environments.

Currently, XDoc is primarily text-centric and does not incorporate raw visual features (pixel-level data) used by models like LayoutLMv3. Future iterations may integrate visual encoders to better understand document aesthetics and logos. Additionally, the framework could be expanded to include other common formats such as presentation slides and spreadsheets, further unifying the field of automated document processing.

Summary updated

Abstract

The surge of pre-training has witnessed the rapid development of document understanding recently. Pre-training and fine-tuning framework has been effectively used to tackle texts in various formats, including plain texts, document texts, and web texts. Despite achieving promising performance, existing pre-trained models usually target one specific document format at one time, making it difficult to combine knowledge from multiple document formats. To address this, we propose XDoc, a unified pre-trained model which deals with different document formats in a single model. For parameter efficiency, we share backbone parameters for different formats such as the word embedding layer and the Transformer layers. Meanwhile, we introduce adaptive layers with lightweight parameters to enhance the distinction across different formats. Experimental results have demonstrated that with only 36.7% parameters, XDoc achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models, which is cost effective for real-world deployment. The code and pre-trained models will be publicly available at \url{https://aka.ms/xdoc}.
BibTeX
@inproceedings{chen2022xdoc,
  author = {Jingye Chen and Tengchao Lv and Lei Cui and Cha Zhang and Furu Wei},
  title = {{XDoc: Unified Pre-training for Cross-Format Document Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=W10zSjrWt8o},
}
Figure