DiT: Self-supervised Pre-training for Document Image Transformer

Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

TL;DR

DiT (Document Image Transformer) is a self-supervised framework that pre-trains vision Transformers specifically for Document AI. By leveraging 42 million unlabeled document pages and a domain-specific image tokenizer, DiT bridges the gap between natural image processing and complex document analysis, setting new state-of-the-art records across classification, layout analysis, and table detection.

What Problem Does This Paper Address?

Most Document AI systems rely on "backbones"—the underlying neural networks that see the image—trained on datasets like ImageNet. While ImageNet is great for identifying cats and dogs, document images are fundamentally different: they are composed of lines, text, tables, and specific hierarchical layouts.

The challenge is that we don't have a "Document ImageNet" with millions of human-labeled categories to train these backbones from scratch. Consequently, researchers often face a "domain shift" where models trained on natural scenes struggle with the fine-grained structures of a tax form or a scientific paper. DiT addresses this by proving that we don't need labels; we just need a massive amount of unlabeled data and the right self-supervised "pretext" task.

Methods

DiT adopts the Masked Image Modeling (MIM) strategy, popularized by models like BEiT. The process works in two main stages:

  1. The Document-Specific Tokenizer: Standard models use a tokenizer (a discrete VAE) trained on natural images (DALL-E). The authors found this "blurry" for documents. They instead trained a custom dVAE on 42 million document pages. This allows the model to represent document patches as discrete visual tokens that preserve sharp edges and text structures.
  2. Self-Supervised Pre-training: The model takes a document image, splits it into patches, and randomly masks a portion of them. The Transformer (DiT) must then predict the visual tokens of the missing patches based on the surrounding context.

By forcing the model to "fill in the blanks" of a masked document, it naturally learns the global relationship between headers, paragraphs, tables, and figures without ever seeing a single human label during pre-training.

Datasets

  • Pre-training: The IIT-CDIP Test Collection 1.0, which contains roughly 42 million scanned document pages.
  • Evaluation (Fine-tuning):
    • RVL-CDIP: 400,000 images for document image classification (16 categories like "letter," "invoice," "memo").
  • PubLayNet: A large-scale dataset for document layout analysis (detecting text, titles, lists, figures, and tables).
  • ICDAR 2019 cTDaR: A benchmark for table detection in both modern and archival documents.
  • FUNSD: A noisy scanned document dataset used here for OCR text detection.

Key Insights and Ideas

  • Domain-Specific Tokenization is Crucial: The "Aha!" moment comes from Figure 4 in the paper. When reconstructing a document image, the standard DALL-E tokenizer produces gibberish, while the DiT-trained dVAE produces legible text and sharp lines. This high-fidelity representation is the foundation of DiT's success.
  • Scaling Matters: The authors released both a Base (87M parameters) and a Large (304M parameters) version. The Large model consistently outperformed the Base model, suggesting that document understanding scales well with model capacity.
  • Universal Backbone: Unlike previous models that required joint training of text, layout, and image features, DiT proves that a pure vision backbone, if pre-trained correctly, can significantly boost performance even before adding language components.

Key Results

DiT achieved new State-of-the-Art (SOTA) results across all tested benchmarks:

  • Classification (RVL-CDIP): Reached 92.69% accuracy, outperforming previous SOTA models like MarkupLM and even ensemble models.
  • Layout Analysis (PubLayNet): Achieved an mAP of 94.9, significantly higher than the previous 91.0.
  • Table Detection (ICDAR 2019): Improved the weighted F1 score from 94.23 to 96.55.
  • Text Detection (FUNSD): Achieved an F1 score of 94.29, proving its utility for downstream OCR tasks.

Limitations and Future Work

  • Archival Complexity: While DiT performed exceptionally well on modern documents, the authors noted that archival documents (which are often handwritten or degraded) required specific pre-processing like adaptive binarization to reach peak performance.
  • Grayscale vs. Color: The current tokenizer was trained largely on grayscale images. Future iterations might benefit from better color representation for documents with complex colored graphics or stamps.
  • Multimodal Integration: The authors suggest that the next step is integrating DiT as the vision component in multimodal models (like LayoutLM), combining its visual prowess with natural language processing for a "complete" document understanding system.

Summary updated

Abstract

Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human-labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, table detection as well as text detection for OCR. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 - 92.69), document layout analysis (91.0 - 94.9), table detection (94.23 - 96.55) and text detection for OCR (93.07 - 94.29). The code and pre-trained models are publicly available at https://aka.ms/msdit.
BibTeX
@inproceedings{li2022dit,
  author = {Junlong Li and Yiheng Xu and Tengchao Lv and Lei Cui and Cha Zhang and Furu Wei},
  title = {{DiT: Self-supervised Pre-training for Document Image Transformer}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=9aBEEAglewc},
}

Loading related content...

Figure