Introduction
Classifying semi-structured documents, such as invoices, forms, and resumes, presents a significant challenge for traditional natural language processing (NLP) models. Standard classification approaches typically operate under a "closed-world" assumption, where the categories are fixed and require extensive labeled data for each class. However, in practical applications, new document types frequently emerge, necessitating a more flexible, zero-shot approach.
Furthermore, semi-structured documents possess a two-dimensional nature where spatial layout is as critical as textual content. Standard models like BERT often fail to capture these visual relationships, leading to suboptimal performance on documents where the physical position of text blocks conveys essential semantic information.
Methods
To address these challenges, we propose a matching-based framework utilizing a dual-encoder architecture.
Architecture
- Document Encoder ($\Phi_{doc}$): We utilize LayoutBERT, a 6-layer Transformer designed to process both textual tokens and their corresponding 2D coordinates (bounding box centroids).
- Label Encoder ($\Phi_{label}$): A standard BERT-base model is employed to encode the natural language names of document classes.
Unsupervised Contrastive Pretraining (CPT)
The core innovation of this work is the Unsupervised Contrastive Pretraining (CPT) phase. This self-supervised step "warms up" the encoders before exposure to labeled data:
- Pseudo-labels: A continuous block of text is sampled from a raw document to act as a "pseudo-label."
- Contrastive Loss: The model is trained to maximize the similarity (dot product) between a document's representation and its own pseudo-label, while minimizing similarity with pseudo-labels from other documents in the same batch.
For supervised zero-shot scenarios, the model is further fine-tuned using the same contrastive objective, replacing pseudo-labels with actual ground-truth class names.
Experimental Setup
The framework was evaluated using the following datasets:
- RVL-CDIP: A benchmark containing 400,000 grayscale document images across 16 categories. We established four "zero-shot splits," training on 8 classes and testing on 4 entirely unseen classes.
- CommonCrawl: For unsupervised pretraining, we utilized 2.3 million PDF documents (approximately 850 million tokens) to capture a wide variety of layouts and text structures.
Key Results and Insights
Our evaluation demonstrates that the combination of layout-aware encoding and contrastive pretraining significantly enhances zero-shot performance.
The Importance of Layout Information
Replacing a standard BERT document encoder with LayoutBERT yielded an average F1 score improvement of approximately 8 points. This confirms that spatial positioning is a primary signal for identifying semi-structured document types.
Impact of Contrastive Pretraining
CPT proved to be the most impactful component, providing a ~19 point F1 boost in unsupervised settings and a ~9 point boost in supervised settings. This suggests that learning the general relationship between content blocks and document context is a powerful precursor to classification.
The Supervision Paradox
Interestingly, the unsupervised zero-shot model occasionally outperformed its supervised counterpart. This "Supervision Paradox" suggests that fine-tuning on specific training classes may cause the model to overfit to those particular document styles, potentially hindering its ability to generalize to entirely new document types during inference.
Efficiency
LayoutBERT maintains high effectiveness while being significantly leaner than models like LayoutLM. By using only 6 layers and omitting heavy image-based CNN components, it offers a more efficient solution for document classification tasks.
Limitations and Future Work
While promising, this research has limitations that warrant further investigation. The results are currently based on the RVL-CDIP benchmark; future work should explore generalization across more diverse or specialized domains. Additionally, integrating more sophisticated multi-modal encoders that incorporate raw image features could further improve performance. Finally, the observed "Supervision Paradox" indicates a need for advanced meta-learning techniques to help supervised models generalize more effectively to unseen classes.
Summary updated