AliGATr: Graph-based layout generation for form understanding

Armineh Nourbakhsh, Zhao Jin, Siddharth Parekh, Sameena Shah, Carolyn P. RoséEMNLP 2024 · DOI

Introduction

Understanding structured documents like invoices, tax forms, and medical records remains a significant challenge in document AI. This task, known as form understanding, requires not only identifying specific fields (Key Information Extraction, or KIE) but also establishing the semantic links between them (Relation Extraction, or RE). While Transformer-based models like LayoutLM have set the standard, they often suffer from high computational costs, a requirement for massive pre-training datasets, and poor probability calibration.

AliGATr addresses these issues by shifting the paradigm from sequential or patch-based processing to a graph-based approach. By leveraging the inherent grid-like structure of forms, AliGATr achieves state-of-the-art performance with significantly fewer parameters and less data.

The AligNet Graph Representation

A core innovation of this work is AligNet, a novel graph construction method designed specifically for document layouts. Traditional graph-based models often rely on K-Nearest Neighbors (KNN) or Line-of-Sight (LOS) connections, which can be noisy or fail to capture long-range structural dependencies.

AligNet utilizes "soft alignments" to create edges. If two tokens are horizontally or vertically aligned within a specific threshold, an edge is established. This approach effectively captures the "skeleton" of a form—where headers align with columns and keys align with their corresponding values—providing a much stronger inductive bias than simple spatial proximity.

Comparison of graph construction methods for document layouts. AligNet (f) focuses on structural alignment rather than simple proximity.

Model Architecture and Generative Pre-training

AliGATr is built on a Graph Neural Network (GNN) using a Relational Graph Attention (RGAT) backbone. Unlike many models that rely on masked language modeling, AliGATr introduces a generative layout objective during pre-training:

  1. Next-Node Prediction: The model learns to select the next token in a reading sequence from a set of candidates.
  2. Adjacency Prediction: Once a node is selected, the model predicts its connections to previously identified nodes.
  3. Layout Generation: By reconstructing the document layout token-by-token, the model develops a deep understanding of spatial relationships without the need for the massive text-generation datasets required by Large Language Models (LLMs).

This generative approach has a secondary benefit: it significantly improves model calibration. While standard Transformers tend to be over-confident (often reporting 99% confidence even when incorrect), AliGATr’s predicted probabilities more accurately reflect its actual performance, making it more reliable for industrial applications.

Results and Efficiency

AliGATr was evaluated on several standard benchmarks, including FUNSD, SROIE, CORD, and BuDDIE. Despite being pre-trained on only 1 million documents—an 11x reduction compared to the 11 million typically used for LayoutLM—AliGATr delivered competitive or superior results.

Key highlights include:

  • Parameter Efficiency: At 145M parameters, AliGATr is 30% smaller than FormNetv2 and significantly more compact than LayoutLMv3 (357M).
  • Superior Relation Extraction: On the FUNSD dataset, AliGATr achieved an 89.50 F1 score for RE, a substantial improvement over LayoutLMv3’s 80.35.
  • Reliable Calibration: The model demonstrated a lower Expected Calibration Error (ECE), providing a more trustworthy measure of uncertainty.

Conclusion and Future Directions

AliGATr demonstrates that incorporating structural inductive biases through graph representations can lead to more efficient and reliable document understanding models. By focusing on alignment rather than proximity and using a generative layout objective, the model overcomes many of the limitations of current Transformer-based architectures.

Future work will focus on improving robustness to OCR errors, as AligNet’s reliance on precise alignment makes it sensitive to document skew. Additionally, researchers aim to extend these graph-based methods to multi-column layouts and non-English documents to test their generalizability across different linguistic and formatting conventions.

Summary updated

BibTeX
@inproceedings{nourbakhsh2024aligatr,
  author = {Armineh Nourbakhsh and Zhao Jin and Siddharth Parekh and Sameena Shah and Carolyn P. Rosé},
  title = {{AliGATr: Graph-based layout generation for form understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.778},
}
Figure