TL;DR
FS-DAG is a lightweight, graph-based framework for Visually Rich Document Understanding (VRDU) that delivers state-of-the-art information extraction performance using fewer than 90 million parameters. By combining pluggable text/vision backbones with a Graph Neural Network (GNN), it adapts to new document domains with as few as five training examples and maintains remarkable immunity to OCR errors.
What Problem Does This Paper Address?
Visually Rich Document Understanding (VRDU) is crucial for automating data extraction from invoices, medical forms, and bank statements. While Large Multimodal Models (LMMs) and architectures like LayoutLM excel at these tasks, they are computationally heavy, slow, and require massive amounts of labeled data to adapt to new document types.
In real-world industrial settings, data scarcity is the norm—companies often need to support a new document format with only a handful of examples (few-shot learning). Furthermore, production pipelines rely on Optical Character Recognition (OCR) engines, which frequently introduce misspellings, character substitutions, and layout shifts. Monolithic transformer models heavily rely on exact text sequences and tokenization; when the OCR engine fails, the model's performance plummets. The authors introduce FS-DAG to bridge this gap by creating a small, fast, and OCR-robust model tailored for few-shot domain adaptation.
Methods
FS-DAG shifts away from monolithic architectures toward a modular, graph-based approach.
As illustrated in Figure 1, the pipeline processes a document image through the following steps:
- Feature Extraction: An OCR engine extracts text regions ($r_i$), which are clustered and sorted into a consistent reading sequence ($s$). Textual features ($t_i$) are extracted using a pre-trained language model (like DistilBERT). Concurrently, visual features ($v_i$) are captured by applying ROI-Align over the feature maps of a visual backbone (UNet with a ResNet-18 backbone).
- Deep Fusion: The textual and visual features are fused using a Kronecker product to initialize the graph's node features ($n_i$).
- Graph Neural Network (GNN): The document is modeled as a graph where nodes are text segments and edges represent spatial/semantic relationships. The GNN incorporates learnable, shared horizontal and vertical positional embeddings ($p_i$) and a multi-head attention mechanism to dynamically weight edge connections during message passing.
- Classification: A final classification head categorizes the node features into specific key-value classes.
To stabilize training in low-data regimes, the authors eliminate edge-linking loss, introduce Label Smoothing to the cross-entropy loss, and apply instance normalization over the node features to prevent overfitting.
Datasets
The model was evaluated on both public and newly introduced industry-specific datasets:
- WildReceipt: A public dataset containing restaurant receipts annotated with 25 key-value classes.
- Proprietary Industrial Dataset: Introduced by the authors to reflect complex, fine-grained industrial use cases. It is split into two categories:
- Category 1 (5 document types): Includes Ecommerce Invoices, Adverse Reaction Health Forms, Medical Invoices, University Admission Forms, and Visa Forms.
- Category 2 (7 document types): Features highly structured documents filled character-by-character, such as Personal Bank Accounts, Equity Mortgages, and Medical Tax Returns.
Key Insights and Ideas
- Graphs are inherently more robust to OCR errors than sequences: When an OCR engine misspells a word, a transformer's tokenization changes drastically, disrupting its sequential understanding. Because FS-DAG relies on a GNN that heavily weights spatial layout and visual features alongside text, it preserves context even when individual words are garbled.
- Modularity enables domain specialization: Instead of using a generic, massive model, FS-DAG allows developers to swap out the text backbone for domain-specific models (e.g., FinBERT for financial documents or BioBERT for medical forms). This plug-and-play capability yields significant performance boosts without requiring the model architecture to grow.
- Sub-token pooling beats truncation: When extracting text embeddings for an OCR bounding box, pooling all sub-token representations generated by the language model provides a much richer semantic representation than simply selecting the first sub-token.
Key Results
The authors benchmarked FS-DAG against models like BERT, DistilBERT, SDMGR, LayoutLMv2, and LayoutLMv3 in a strict 5-shot learning setting (training on only 5 documents).
As shown in Table 2 (Figure 3), FS-DAG achieved dominant results:
- Top-Tier Accuracy: It achieved an average F1-score of 98.89% on Category 1 and 99.93% on Category 2 datasets when no OCR errors were present, outperforming LayoutLMv3 despite having 44 million fewer parameters.
- Unmatched OCR Robustness: When synthetic OCR errors were introduced during inference (with a 10% probability), transformer-based models collapsed—LayoutLMv2's performance dropped by 19.46% and BERT's by 25.24%. In contrast, FS-DAG's performance dropped by less than 1% (0.93% and 0.91% respectively).
- Efficiency: With only 81M parameters, FS-DAG cut inference latency to 773 ms, making it roughly twice as fast as LayoutLMv2 (1907 ms) and significantly faster than LayoutLMv3 (1363 ms).
Limitations and Future Work
Currently, the FS-DAG architecture does not support entity-linking (mapping which specific key belongs to which value); it relies solely on node classification. The authors intentionally eliminated the edge loss function to stabilize the model during few-shot training.
For future work, the authors plan to extend FS-DAG's capabilities to zero-shot learning scenarios and adapt the framework to support a broader array of complex industrial layout structures without requiring any fine-tuning data.
Summary updated