DiT: Self-supervised Pre-training for Document Image Transformer

Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

Abstract

Recent advancements in Image Transformers have revolutionized natural image understanding. However, applying these models to Document AI has been hindered by the lack of large-scale labeled datasets. We introduce DiT, a self-supervised pre-trained Document Image Transformer. By leveraging 42 million unlabeled document images, DiT establishes a robust backbone for various Document AI tasks. Our experiments demonstrate that DiT achieves state-of-the-art (SOTA) performance in document image classification, layout analysis, table detection, and text detection, outperforming previous models that relied on natural image pre-training.

Introduction

The backbone of most modern Document AI systems is typically a vision model pre-trained on natural image datasets like ImageNet. While effective for general object recognition, these models often struggle with the unique structural characteristics of documents—such as fine-grained text, complex hierarchical layouts, and tabular data. This "domain shift" limits the performance of standard Vision Transformers (ViT) when applied to professional, technical, or archival documents.

To bridge this gap, we propose DiT (Document Image Transformer). Unlike supervised models that require labor-intensive human labeling, DiT utilizes self-supervised learning on a massive scale. By learning the underlying visual grammar of documents from 42 million unlabeled pages, DiT provides a specialized foundation for downstream Document AI applications.

Methodology: Masked Image Modeling for Documents

DiT utilizes a Masked Image Modeling (MIM) strategy, similar to the BEiT architecture, but optimized specifically for the document domain. The training pipeline consists of two primary components:

1. Document-Specific Tokenization

Standard visual tokenizers (such as the discrete VAE used in DALL-E) are often optimized for natural scenes, which can result in "blurry" reconstructions of text-heavy images. To address this, we trained a custom dVAE on 42 million document pages. This specialized tokenizer preserves the sharp edges and high-frequency details essential for representing text and lines, providing a high-fidelity visual vocabulary for the Transformer.

2. Self-Supervised Pre-training

During pre-training, a document image is partitioned into patches, and a subset of these patches is randomly masked. The DiT model is then tasked with predicting the visual tokens of the masked patches based on the surrounding visible context. This "fill-in-the-blanks" approach forces the model to learn the spatial and structural relationships between different document elements—such as how a header relates to a paragraph or how cell borders define a table—without requiring any human-annotated labels.

Experimental Setup

We pre-trained DiT on the IIT-CDIP Test Collection 1.0, a repository containing approximately 42 million scanned document pages. To evaluate its versatility, we fine-tuned the model on several key benchmarks:

  • RVL-CDIP: Document image classification across 16 categories (e.g., letter, invoice, memo).
  • PubLayNet: Document layout analysis (detecting text, titles, lists, figures, and tables).
  • ICDAR 2019 cTDaR: Table detection in both modern and archival documents.
  • FUNSD: Text detection in noisy, scanned forms for OCR tasks.

Key Results and Insights

DiT consistently outperformed previous state-of-the-art models across all benchmarks, demonstrating the power of domain-specific pre-training.

Performance Summary: * Classification (RVL-CDIP): Achieved 92.69% accuracy, surpassing previous SOTA models like MarkupLM. * Layout Analysis (PubLayNet): Reached an mAP of 94.9, a significant jump from the previous 91.0. * Table Detection (cTDaR): Improved the weighted F1 score to 96.55. * Text Detection (FUNSD): Achieved an F1 score of 94.29.

Beyond these metrics, our analysis revealed that domain-specific tokenization is crucial. While standard tokenizers produce illegible results on document patches, the DiT-trained dVAE maintains the legibility of text and the sharpness of lines. Furthermore, we found that document understanding scales effectively with model capacity; the DiT-Large model (304M parameters) consistently outperformed the DiT-Base model (87M parameters).

Discussion and Future Work

Our findings highlight that a pure vision backbone, when pre-trained on the correct domain, can serve as a universal engine for Document AI. While previous approaches often relied on complex multi-modal fusion (combining text, layout, and image features early on), DiT proves that strong visual representations alone can drive substantial improvements.

However, challenges remain in processing archival documents, which often require adaptive binarization to handle degradation. Future iterations may also benefit from enhanced color representation for documents with complex graphics. The next logical step is the integration of DiT as the visual component in multimodal models (like LayoutLM), combining its visual prowess with natural language processing for a comprehensive document understanding system.

Conclusion

DiT represents a significant step forward in Document AI. By shifting from supervised natural image pre-training to self-supervised document image pre-training, we have provided a model that inherently understands the "language of layouts." The code and pre-trained models are available to the research community to further accelerate progress in automated document understanding.

Summary updated

Abstract

Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose \textbf{DiT}, a self-supervised pre-trained \textbf{D}ocument \textbf{I}mage \textbf{T}ransformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human-labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, table detection as well as text detection for OCR. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 $\rightarrow$ 92.69), document layout analysis (91.0 $\rightarrow$ 94.9), table detection (94.23 $\rightarrow$ 96.55) and text detection for OCR (93.07 $\rightarrow$ 94.29). The code and pre-trained models are publicly available at \url{https://aka.ms/msdit}.
BibTeX
@inproceedings{li2022dit,
  author = {Junlong Li and Yiheng Xu and Tengchao Lv and Lei Cui and Cha Zhang and Furu Wei},
  title = {{DiT: Self-supervised Pre-training for Document Image Transformer}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=9aBEEAglewc},
}
Figure