Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents

Muhammad Khalifa, Yogarshi Vyas, Shuai Wang, Graham Horwood, Sunil Mallya, Miguel Ballesteros

Abstract

We investigate semi-structured document classification in a zero-shot setting. Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents. The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new document categories could potentially emerge. We focus exclusively on the zero-shot setting where inference is done on new unseen classes. To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning. Our results show a significant boost in Macro F$_1$ from the proposed pretraining step in both supervised and unsupervised zero-shot settings.

Introduction

In real-world applications, document categories are rarely static. Organizations frequently encounter new types of documents—such as novel tax forms or specialized shipping manifests—that were not present in the original training data. This necessitates zero-shot classification, where a model must identify a document's category without having seen any labeled examples of that specific class during training.

Semi-structured documents (e.g., invoices, resumes, and forms) present a unique challenge because their visual layout is as informative as their textual content. A "Total Due" field carries different weight depending on its spatial position. Standard NLP models often treat documents as flat text strings, ignoring these critical spatial cues. This paper addresses these challenges by combining layout-aware encoding with a contrastive learning strategy designed for flexibility in dynamic environments.

Methodology

The researchers propose a matching-based framework centered on three key pillars:

  1. LayoutBERT: A streamlined, layout-aware encoder based on the Transformer architecture. Unlike complex models that utilize heavy image-processing (CNN) components, LayoutBERT uses only the centroid (center point) of a word's bounding box to encode 2D positions, keeping the model lightweight and efficient.
  2. Contrastive Layout Pretraining (CPT): This unsupervised strategy "warms up" the model by training it to match a document to a random block of its own text, which acts as a pseudo-label. The model learns to maximize similarity between a document and its internal snippets while minimizing similarity with snippets from other documents.
  3. Two-Tower Architecture: The system employs a dual-encoder setup: one for the document and a separate vanilla BERT encoder for class labels. By projecting both into a shared embedding space, the model can "match" an unseen document to the label it most closely resembles mathematically.
The proposed matching-based framework using a two-tower architecture and contrastive objective.

Experimental Setup

To evaluate the model, the authors utilized two primary data sources:

  • Pretraining: 800,000 documents sampled from a 2.3-million PDF crawl of CommonCrawl were used for the unsupervised contrastive pretraining step.
  • Evaluation: The RVL-CDIP dataset, containing 400,000 images across 16 categories, served as the benchmark.
  • Zero-Shot Splits: To ensure rigorous testing, the authors created four distinct splits. In each, the model was trained on 8 categories, validated on 4, and tested on 4 entirely "unseen" categories.

Results and Discussion

The experimental results demonstrate that Contrastive Layout Pretraining (CPT) significantly outperforms standard baselines.

  • Unsupervised Performance: In settings where no human-labeled data was used, CPT improved the average Macro F1 score from a baseline of 12.61 to 39.63. This highlights the power of pseudo-labels in learning a robust "language of documents."
  • Supervised Zero-Shot: Even when the model was fine-tuned on different labeled categories, CPT provided a substantial boost, raising the average F1 from 28.76 to 37.23.
  • The Importance of Layout: Simply incorporating spatial coordinates (moving from BERT to LayoutBERT) improved scores by approximately 8 points, confirming that layout is a primary signal for document understanding.
Performance comparison in the unsupervised zero-shot setting across four data splits.

An interesting finding was the "Fine-Tuning Trap": in certain cases, unsupervised models outperformed those fine-tuned on specific training classes. This suggests that standard fine-tuning can lead to overfitting on specific styles, reducing the model's flexibility when encountering entirely new document types.

Conclusion

This work demonstrates that contrastive pretraining with pseudo-labels is a highly effective strategy for zero-shot classification of semi-structured documents. While the current approach uses simplified 2D coordinates and ignores visual features like logos or fonts, it provides a strong foundation for future work. Potential improvements include integrating richer document representations and exploring meta-learning techniques to further enhance how models adapt to new document categories on the fly.

Summary updated

Abstract

We investigate semi-structured document classification in a zero-shot setting. Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents. The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new document categories could potentially emerge. We focus exclusively on the zero-shot setting where inference is done on new unseen classes. To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning. Our results show a significant boost in Macro F$_1$ from the proposed pretraining step in both supervised and unsupervised zero-shot settings.
BibTeX
@inproceedings{khalifa2023contrastive,
  author = {Muhammad Khalifa and Yogarshi Vyas and Shuai Wang and Graham Horwood and Sunil Mallya and Miguel Ballesteros},
  title = {{Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=op-tw6Sbvpp},
}
Figure