Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models

Lei Wang, Jiabang He, Xing Xu, Ning Liu, Hui Liu

Abstract

Pre-trained multimodal models have significantly advanced Document AI, yet they often rely on coarse-grained objectives during pre-training and discard alignment mechanisms during fine-tuning. This paper presents AETNet, which introduces Alignment-Enriched Tuning. By incorporating fine-grained image-text alignment objectives—most notably Patch-level Image-Text Alignment (PITA)—directly into the fine-tuning phase, AETNet enhances the model's structural understanding. Experimental results on benchmarks such as FUNSD, CORD, and DocVQA demonstrate that AETNet consistently outperforms standard fine-tuning baselines, achieving state-of-the-art performance for base-sized models.

Introduction

The current paradigm in Document AI involves pre-training large-scale models like LayoutLMv3 on massive datasets using self-supervised losses. However, these models are typically adapted to downstream tasks via standard supervised fine-tuning, which often ignores the intricate spatial relationships between text and image patches. Document images are inherently complex; the physical location of a word is as important as the word itself.

Comparison of different tuning paradigms: Fine-Tuning, Prompt-Tuning, and Alignment-Enriched Tuning.

The authors of AETNet argue that maintaining and even refining these alignment objectives during fine-tuning can serve as a powerful regularizer. They propose a method to "bake" sophisticated alignment objectives into the tuning process, avoiding the prohibitive costs of re-pre-training while capturing finer-grained document structures.

Methodology: The AETNet Framework

AETNet modifies the standard fine-tuning architecture by introducing additional components and a multi-faceted objective function.

Architecture Enhancements

  1. Alignment-Aware Encoders: AETNet adds an auxiliary Vision Transformer (ViT) and a Text Transformer (RoBERTa). These encoders process modalities independently before fusion in the main multimodal backbone, allowing for explicit cross-modal alignment.
  2. Momentum Modeling: To stabilize the contrastive learning process, the framework utilizes momentum-based moving average versions of the encoders and maintains memory queues for recent samples.

Multi-Part Objective Function

The model is optimized using a joint loss function comprising five distinct objectives:

  • Supervised Objective (SO): The primary task-specific loss.
  • Document-Level Contrastive (DITC): Aligns global image and text representations.
  • Intra-Modal Contrastive (IMC): Enhances feature robustness within individual modalities.
  • Global-Local Contrastive (GLITC): Models the relationship between local components (patches/tokens) and the entire document.
  • Patch-Level Image-Text Alignment (PITA): The core innovation. PITA leverages OCR layout information to explicitly align specific text tokens with their corresponding image patches.
Detailed architecture of AETNet showcasing the integration of various contrastive objectives including PITA.

Experimental Results

AETNet was evaluated across several key Document AI benchmarks, showing consistent improvements over strong baselines:

  • FUNSD (Form Understanding): Achieved an F1 score of 91.55, a notable increase from the LayoutLMv3 baseline of 89.82.
  • CORD (Receipt Extraction): Reached 97.04 F1, demonstrating superior key information extraction capabilities.
  • DocVQA: Improved accuracy to 79.73 ANLS, setting a new benchmark for base-sized models.
  • RVL-CDIP-1: In classification tasks, AETNet outperformed both standard fine-tuning and prompt-tuning methods.

Discussion and Insights

The primary takeaway is that alignment is not just for pre-training. By treating self-supervised style losses as regularizers during supervised tuning, AETNet maintains a grounded understanding of document structure. The PITA loss is particularly effective, as it forces the model to learn "what is where" at a granular level. Case studies indicate that AETNet is better at resolving layout ambiguities that often confuse vanilla models, such as distinguishing between "Question" and "Header" fields in complex forms.

Conclusion

AETNet provides a computationally efficient way to enhance pre-trained document models. By enriching the tuning process with fine-grained alignment objectives, it bridges the gap between coarse pre-training and the high-precision requirements of downstream document understanding tasks. Future work may explore applying these techniques to larger generative models or extremely low-resource scenarios.

Summary updated

Abstract

Alignment between image and text has shown promising improvements on patch-level pre-trained document image models. However, investigating more effective or finer-grained alignment techniques during pre-training requires a large amount of computation cost and time. Thus, a question naturally arises: Could we fine-tune the pre-trained models adaptive to downstream tasks with alignment objectives and achieve comparable or better performance? In this paper, we propose a new model architecture with alignment-enriched tuning (dubbed AETNet) upon pre-trained document image models, to adapt downstream tasks with the joint task-specific supervised and alignment-aware contrastive objective. Specifically, we introduce an extra visual transformer as the alignment-ware image encoder and an extra text transformer as the alignment-ware text encoder before multimodal fusion. We consider alignment in the following three aspects: 1) document-level alignment by leveraging the cross-modal and intra-modal contrastive loss; 2) global-local alignment for modeling localized and structural information in document images; and 3) local-level alignment for more accurate patch-level information. Experiments on various downstream tasks show that AETNet can achieve state-of-the-art performance on various downstream tasks. Notably, AETNet consistently outperforms state-of-the-art pre-trained models, such as LayoutLMv3 with fine-tuning techniques, on three different downstream tasks. Code is available at https://github.com/MAEHCM/AET.
BibTeX
@inproceedings{wang2023alignment,
  author = {Lei Wang and Jiabang He and Xing Xu and Ning Liu and Hui Liu},
  title = {{Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25357},
}
Figure