Abstract
Pre-trained multimodal models have significantly advanced Document AI, yet they often rely on coarse-grained objectives during pre-training and discard alignment mechanisms during fine-tuning. This paper presents AETNet, which introduces Alignment-Enriched Tuning. By incorporating fine-grained image-text alignment objectives—most notably Patch-level Image-Text Alignment (PITA)—directly into the fine-tuning phase, AETNet enhances the model's structural understanding. Experimental results on benchmarks such as FUNSD, CORD, and DocVQA demonstrate that AETNet consistently outperforms standard fine-tuning baselines, achieving state-of-the-art performance for base-sized models.
Introduction
The current paradigm in Document AI involves pre-training large-scale models like LayoutLMv3 on massive datasets using self-supervised losses. However, these models are typically adapted to downstream tasks via standard supervised fine-tuning, which often ignores the intricate spatial relationships between text and image patches. Document images are inherently complex; the physical location of a word is as important as the word itself.
The authors of AETNet argue that maintaining and even refining these alignment objectives during fine-tuning can serve as a powerful regularizer. They propose a method to "bake" sophisticated alignment objectives into the tuning process, avoiding the prohibitive costs of re-pre-training while capturing finer-grained document structures.
Methodology: The AETNet Framework
AETNet modifies the standard fine-tuning architecture by introducing additional components and a multi-faceted objective function.
Architecture Enhancements
- Alignment-Aware Encoders: AETNet adds an auxiliary Vision Transformer (ViT) and a Text Transformer (RoBERTa). These encoders process modalities independently before fusion in the main multimodal backbone, allowing for explicit cross-modal alignment.
- Momentum Modeling: To stabilize the contrastive learning process, the framework utilizes momentum-based moving average versions of the encoders and maintains memory queues for recent samples.
Multi-Part Objective Function
The model is optimized using a joint loss function comprising five distinct objectives:
- Supervised Objective (SO): The primary task-specific loss.
- Document-Level Contrastive (DITC): Aligns global image and text representations.
- Intra-Modal Contrastive (IMC): Enhances feature robustness within individual modalities.
- Global-Local Contrastive (GLITC): Models the relationship between local components (patches/tokens) and the entire document.
- Patch-Level Image-Text Alignment (PITA): The core innovation. PITA leverages OCR layout information to explicitly align specific text tokens with their corresponding image patches.
Experimental Results
AETNet was evaluated across several key Document AI benchmarks, showing consistent improvements over strong baselines:
- FUNSD (Form Understanding): Achieved an F1 score of 91.55, a notable increase from the LayoutLMv3 baseline of 89.82.
- CORD (Receipt Extraction): Reached 97.04 F1, demonstrating superior key information extraction capabilities.
- DocVQA: Improved accuracy to 79.73 ANLS, setting a new benchmark for base-sized models.
- RVL-CDIP-1: In classification tasks, AETNet outperformed both standard fine-tuning and prompt-tuning methods.
Discussion and Insights
The primary takeaway is that alignment is not just for pre-training. By treating self-supervised style losses as regularizers during supervised tuning, AETNet maintains a grounded understanding of document structure. The PITA loss is particularly effective, as it forces the model to learn "what is where" at a granular level. Case studies indicate that AETNet is better at resolving layout ambiguities that often confuse vanilla models, such as distinguishing between "Question" and "Header" fields in complex forms.
Conclusion
AETNet provides a computationally efficient way to enhance pre-trained document models. By enriching the tuning process with fine-grained alignment objectives, it bridges the gap between coarse pre-training and the high-precision requirements of downstream document understanding tasks. Future work may explore applying these techniques to larger generative models or extremely low-resource scenarios.
Summary updated