Abstract
In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable and efficient model architecture for visually rich document understanding (VRDU) in few-shot settings. FS-DAG leverages domain-specific and language/vision specific backbones within a modular framework to adapt to diverse document types with minimal data. The model is robust to practical challenges such as handling OCR errors, misspellings, and domain shifts, which are critical in real-world deployments. FS-DAG is highly performant with less than 90M parameters, making it well-suited for complex real-world applications for Information Extraction (IE) tasks where computational resources are limited. We demonstrate FS-DAG's capability through extensive experiments for information extraction task, showing significant improvements in convergence speed and performance compared to state-of-the-art methods.
Introduction
Visually Rich Document Understanding (VRDU) involves extracting structured information from documents where both textual content and spatial layout are vital, such as invoices, medical forms, and bank statements. While Large Multimodal Models (LMMs) have advanced the state-of-the-art, they present significant hurdles for industrial deployment. Specifically, they often require massive labeled datasets for domain adaptation and entail high computational costs due to their size. Furthermore, many existing models are brittle, where minor OCR errors can lead to catastrophic failures in extraction.
To address these challenges, we present FS-DAG (Few Shot Domain Adapting Graph), a lightweight (81M parameter) framework designed for high-performance information extraction using only a handful of training examples. FS-DAG is built to be modular, efficient, and remarkably resilient to the noise typical of real-world document processing.
Methods
FS-DAG conceptualizes a document as a graph structure where nodes represent individual text regions and edges represent the spatial and semantic relationships between them. This approach allows the model to perform sophisticated spatial reasoning more efficiently than traditional transformer-based sequence models.
Modular Architecture
A core innovation of FS-DAG is its "pluggable" backbone system. Unlike monolithic architectures, FS-DAG allows practitioners to swap the base language model (e.g., Distill-BERT) for domain-specific alternatives (e.g., FinBERT for financial documents) without requiring a total system redesign. This modularity facilitates rapid adaptation to specialized industries.
Feature Fusion and Edge Learning
Node features are initialized through "Deep Fusion," which combines textual embeddings with visual features extracted via a UNET or ResNet-18 backbone using a Kronecker product. This ensures a rich, multi-modal representation for every text region. Additionally, the model employs dynamic edge learning using multi-head attention and shared positional embeddings. This mechanism enables the model to learn both absolute positioning and relative "neighborhood" contexts within the document layout.
Results and Discussion
The model was rigorously evaluated against public benchmarks like WildReceipt and various industrial datasets categorized by complexity.
Performance and Efficiency
On the WildReceipt benchmark, FS-DAG achieved a 93.90% F1-score, significantly outperforming larger models such as LayoutLMv3 (87.14%). In industrial "Category 1" datasets, the model reached an average F1 of 98.89% with only five training samples. Beyond accuracy, FS-DAG demonstrates superior efficiency. With an average inference time of 773ms, it is approximately 43% faster than LayoutLMv3. Training convergence is also rapid, requiring only 21 minutes on average.
Robustness to OCR Noise
One of the most critical findings is the model's resilience to OCR errors. In experiments where OCR errors were introduced with a 10% probability, FS-DAG's performance dropped by less than 1%. In contrast, BERT-based models experienced performance degradations exceeding 25%, highlighting the advantage of FS-DAG's graph-based spatial reasoning in noisy environments.
Conclusion
FS-DAG represents a significant step toward efficient and robust document understanding. By leveraging a modular graph-based architecture, it achieves state-of-the-art performance in few-shot scenarios while maintaining a small computational footprint. Future research will focus on extending the model's capabilities to entity linking and exploring zero-shot learning to further reduce the need for labeled data in new domains.
Summary updated