Introduction
Document understanding is a cornerstone of modern Natural Language Processing (NLP), yet the field remains highly fragmented. Currently, practitioners must deploy specialized models for different document formats: RoBERTa for plain text, LayoutLM for scanned documents, and MarkupLM for web pages. This fragmentation leads to two primary issues: knowledge silos, where models cannot share conceptual similarities across formats, and resource inefficiency, as maintaining multiple large-scale models is computationally expensive.
XDoc addresses these challenges by introducing a unified pre-training framework capable of mastering plain text, document images (PDFs/scans), and web pages within a single architecture. By leveraging a shared Transformer backbone and lightweight adaptive layers, XDoc achieves state-of-the-art performance while significantly reducing the total parameter footprint.
Unified Architecture
The core innovation of XDoc is its ability to process diverse formats through a single, shared backbone. The authors observed that in specialized models, over 90% of the parameters reside in the word embedding and Transformer layers. XDoc capitalizes on this by sharing these components across all formats, facilitating the learning of universal language representations.
Format-Specific Inputs and Adaptive Layers
To accommodate the unique structural characteristics of each format, XDoc utilizes specialized embedding heads:
- Plain Text: Employs standard 1D position embeddings.
- Document Text: Integrates 2D coordinates (bounding boxes) to capture visual layout.
- Web Text: Uses XPath strings to represent the hierarchical DOM structure of HTML.
Rather than simply summing these embeddings, XDoc introduces Adaptive Layers—lightweight Linear-ReLU-Linear sequences. These layers act as specialized translators, processing format-specific information before it enters the shared Transformer backbone. This approach provides the necessary distinction between formats while maintaining a universal backbone, adding only 2.7% to the total parameter count.
Experimental Setup
XDoc was pre-trained using Masked Language Modeling (MLM) on a massive, balanced corpus:
- Plain Text: 213,713 files from Wikipedia, BookCorpus, and OpenWebText.
- Document Text: 11 million scanned pages from the IIT-CDIP collection.
- Web Text: 24 million web pages from Common Crawl.
The model was evaluated on industry-standard benchmarks, including GLUE and SQuAD for plain text, FUNSD and DocVQA for document understanding, and WebSRC for web-based tasks.
Key Results and Insights
XDoc demonstrates that a "jack-of-all-trades" model can indeed be a master of all. Training on a mixture of documents and web pages not only maintains but often improves performance on plain text tasks.
- Parameter Efficiency: A single XDoc model (146M parameters) replaces three specialized models totaling 398M parameters—a 63.3% reduction in total footprint.
- Superior Performance: XDoc achieved an 89.4 F1 score on the FUNSD task, significantly outperforming the specialized LayoutLM baseline (79.3). It also reached 74.8 F1 on WebSRC and 72.7 ANLS on DocVQA.
- Balanced Training: Empirical results suggest that a 1:1:1 balanced ratio of the three document types during pre-training is critical for optimal average performance across all downstream tasks.
Conclusion and Future Work
XDoc represents a significant step toward universal document understanding. While highly effective, the current model relies on metadata (coordinates/XPaths) rather than raw pixels. Future work could involve incorporating image features to bridge the "vision gap" or exploring more complex structural pre-training tasks. Additionally, expanding the framework to include formats like Microsoft Word or PowerPoint remains an open area for research.
Summary updated