FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding

Amit Agarwal, Srikant Panda, Kulbhushan Pachauri

TL;DR

FS-DAG is a lightweight, graph-based framework for Visually Rich Document Understanding (VRDU) that delivers state-of-the-art information extraction performance using fewer than 90 million parameters. By combining pluggable text/vision backbones with a Graph Neural Network (GNN), it adapts to new document domains with as few as five training examples and maintains remarkable immunity to OCR errors.

What Problem Does This Paper Address?

Visually Rich Document Understanding (VRDU) is crucial for automating data extraction from invoices, medical forms, and bank statements. While Large Multimodal Models (LMMs) and architectures like LayoutLM excel at these tasks, they are computationally heavy, slow, and require massive amounts of labeled data to adapt to new document types.

In real-world industrial settings, data scarcity is the norm—companies often need to support a new document format with only a handful of examples (few-shot learning). Furthermore, production pipelines rely on Optical Character Recognition (OCR) engines, which frequently introduce misspellings, character substitutions, and layout shifts. Monolithic transformer models heavily rely on exact text sequences and tokenization; when the OCR engine fails, the model's performance plummets. The authors introduce FS-DAG to bridge this gap by creating a small, fast, and OCR-robust model tailored for few-shot domain adaptation.

Methods

FS-DAG shifts away from monolithic architectures toward a modular, graph-based approach.

Figure 1: An illustration of the model architecture for FS-DAG. Given a document image (I); its text regions {ri} are extracted using an OCR engine. We cluster and sort the {ri} to create a reading sequence {s}; tex- tual features {ti} are extracted using a linear projection layer on top of a pre-trained language model process- ing {s}. In contrast, visual features {vi} are extracted using ROI-Align on top of the feature map from the Visual Model and {ri}. The deep fusion module uses Kronecker product to fuse {ti} and {vi} to initialize the node features {ni}. The node features are propagated and aggregated in the GNN during the message passing, which uses positional embedding {pi} and multi-head attention to learn the edge features dynamically. The classification head will finally classify the node features into one of the key-value classes.

As illustrated in Figure 1, the pipeline processes a document image through the following steps:

  1. Feature Extraction: An OCR engine extracts text regions ($r_i$), which are clustered and sorted into a consistent reading sequence ($s$). Textual features ($t_i$) are extracted using a pre-trained language model (like DistilBERT). Concurrently, visual features ($v_i$) are captured by applying ROI-Align over the feature maps of a visual backbone (UNet with a ResNet-18 backbone).
  2. Deep Fusion: The textual and visual features are fused using a Kronecker product to initialize the graph's node features ($n_i$).
  3. Graph Neural Network (GNN): The document is modeled as a graph where nodes are text segments and edges represent spatial/semantic relationships. The GNN incorporates learnable, shared horizontal and vertical positional embeddings ($p_i$) and a multi-head attention mechanism to dynamically weight edge connections during message passing.
  4. Classification: A final classification head categorizes the node features into specific key-value classes.

To stabilize training in low-data regimes, the authors eliminate edge-linking loss, introduce Label Smoothing to the cross-entropy loss, and apply instance normalization over the node features to prevent overfitting.

Datasets

The model was evaluated on both public and newly introduced industry-specific datasets:

  • WildReceipt: A public dataset containing restaurant receipts annotated with 25 key-value classes.
  • Proprietary Industrial Dataset: Introduced by the authors to reflect complex, fine-grained industrial use cases. It is split into two categories:
  • Category 1 (5 document types): Includes Ecommerce Invoices, Adverse Reaction Health Forms, Medical Invoices, University Admission Forms, and Visa Forms.
  • Category 2 (7 document types): Features highly structured documents filled character-by-character, such as Personal Bank Accounts, Equity Mortgages, and Medical Tax Returns.

Key Insights and Ideas

  • Graphs are inherently more robust to OCR errors than sequences: When an OCR engine misspells a word, a transformer's tokenization changes drastically, disrupting its sequential understanding. Because FS-DAG relies on a GNN that heavily weights spatial layout and visual features alongside text, it preserves context even when individual words are garbled.
  • Modularity enables domain specialization: Instead of using a generic, massive model, FS-DAG allows developers to swap out the text backbone for domain-specific models (e.g., FinBERT for financial documents or BioBERT for medical forms). This plug-and-play capability yields significant performance boosts without requiring the model architecture to grow.
  • Sub-token pooling beats truncation: When extracting text embeddings for an OCR bounding box, pooling all sub-token representations generated by the language model provides a much richer semantic representation than simply selecting the first sub-token.

Key Results

The authors benchmarked FS-DAG against models like BERT, DistilBERT, SDMGR, LayoutLMv2, and LayoutLMv3 in a strict 5-shot learning setting (training on only 5 documents).

Summary of model complexity, performance, robustness, and computational efficiency.

As shown in Table 2 (Figure 3), FS-DAG achieved dominant results:

  • Top-Tier Accuracy: It achieved an average F1-score of 98.89% on Category 1 and 99.93% on Category 2 datasets when no OCR errors were present, outperforming LayoutLMv3 despite having 44 million fewer parameters.
  • Unmatched OCR Robustness: When synthetic OCR errors were introduced during inference (with a 10% probability), transformer-based models collapsed—LayoutLMv2's performance dropped by 19.46% and BERT's by 25.24%. In contrast, FS-DAG's performance dropped by less than 1% (0.93% and 0.91% respectively).
  • Efficiency: With only 81M parameters, FS-DAG cut inference latency to 773 ms, making it roughly twice as fast as LayoutLMv2 (1907 ms) and significantly faster than LayoutLMv3 (1363 ms).

Limitations and Future Work

Currently, the FS-DAG architecture does not support entity-linking (mapping which specific key belongs to which value); it relies solely on node classification. The authors intentionally eliminated the edge loss function to stabilize the model during few-shot training.

For future work, the authors plan to extend FS-DAG's capabilities to zero-shot learning scenarios and adapt the framework to support a broader array of complex industrial layout structures without requiring any fine-tuning data.

Summary updated

Abstract

In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable and efficient model architecture for visually rich document understanding (VRDU) in few-shot settings. FS-DAG leverages domain-specific and language/vision specific backbones within a modular framework to adapt to diverse document types with minimal data. The model is robust to practical challenges such as handling OCR errors, misspellings, and domain shifts, which are critical in real-world deployments. FS-DAG is highly performant with less than 90M parameters, making it well-suited for complex real-world applications for Information Extraction (IE) tasks where computational resources are limited. We demonstrate FS-DAG's capability through extensive experiments for information extraction task, showing significant improvements in convergence speed and performance compared to state-of-the-art methods. Additionally, this work highlights the ongoing progress in developing smaller, more efficient models that do not compromise on performance. Code : https://github.com/oracle-samples/fs-dag
BibTeX
@inproceedings{agarwal2025fs,
  author = {Amit Agarwal and Srikant Panda and Kulbhushan Pachauri},
  title = {{FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=U8WI7H5mll},
}
Figure