Graph Neural Networks for End-to-End Information Extraction From Handwritten Documents

Yessine Khanfir, Marwa Dhiaf, Emna Ghodhbani, Ahmed Cheikh Rouhou, Yousri KessentiniWACV 2024

TL;DR

This paper introduces a novel end-to-end model that jointly performs Handwritten Text Recognition (HTR) and Named Entity Recognition (NER) on historical documents. By combining Graph Neural Networks (GNNs) with Transformer architectures, the model achieves state-of-the-art performance on challenging datasets, demonstrating improved context understanding and robustness compared to previous methods.

What Problem Does This Paper Address?

Extracting structured information from handwritten documents is a significant challenge due to variations in handwriting, noise, and the scarcity of labeled data. Traditional approaches often rely on a two-stage process: first recognizing the text, then extracting named entities. This pipeline is prone to cascading errors, where mistakes in the initial text recognition propagate and negatively impact entity extraction. While end-to-end methods have emerged, they often struggle with capturing complex contextual relationships within documents, especially at a paragraph level. This paper aims to bridge this gap by proposing a unified model that simultaneously handles text recognition and named entity extraction, leveraging graph structures to better understand the relationships between different parts of the document.

Methods

The proposed architecture is an encoder-decoder model that integrates Transformers and Graph Convolutional Networks (GCNs).

  • Encoder: Sparse Graph Transformer Encoder (SGTE): This component is a modified Graph Transformer designed to process visual features from document images. Instead of a fully connected graph, it constructs a sparse graph, where node connections are dynamically defined based on their spatial proximity. This sparsity reduces computational cost while allowing the model to focus attention on relevant parts of the input.
  • Decoder: Cross-GCN-based Decoder: The decoder builds upon the standard Transformer decoder but enhances it with a two-layer GCN. This "Cross-GCN" takes inputs from both the SGTE's output and the Transformer's Multi-Head Attention (MHA) block. It aims to reinforce the alignment between visual features, recognized characters, and Named Entities (NEs) by explicitly modeling relationships across these different representations.

The SGTE and Cross-GCN-based Decoder are trained jointly using a categorical cross-entropy loss.

Datasets

The model was evaluated on two datasets:

  • Esposalles: A dataset of 125 handwritten marriage records, used for information extraction. It was split into 872 records for training, 96 for validation, and 253 for testing. The evaluation considered both a "basic" track (4 named entities) and a "complete" track (26 named entities).
  • IAM Dataset: A benchmark dataset for IE on handwritten documents, consisting of 13353 text lines and 115320 words. The traditional RWTH split was used, with writer-independent partitions for training, validation, and testing. The tag set includes 6 categories: Location, Time, Cardinal, NORP, Person, and Organization.

Key Insights and Ideas

  • The Power of Sparse Graphs: By moving from fully connected graphs to sparse graphs in the encoder, the model achieves computational efficiency without sacrificing performance. This suggests that not all pairwise relationships are equally important for context modeling.
  • GNNs for Enhanced Alignment: The Cross-GCN in the decoder significantly improves the alignment between visual features and the predicted text/entity tags. This is crucial for robust information extraction, as it helps the model better associate visual cues with semantic meaning.
  • End-to-End Advantage: The joint learning of HTR and NER proves more effective than sequential approaches, particularly for complex documents where early errors in transcription can be detrimental. The model's ability to handle paragraph-level context is a key differentiator.
  • Contextual Understanding: The combination of Transformer's attention mechanism and GNN's relational modeling allows the model to capture richer contextual information, leading to better performance on tasks with more complex entity sets.

Key Results

  • State-of-the-Art Performance: The proposed model achieved state-of-the-art results on both the Esposalles dataset (ICDAR 2017 IEHHR competition) and the IAM dataset.
  • Esposalles: Achieved a "complete score" of 96.24%, outperforming the baseline transformer by 0.7% and other previous methods. The gap between basic and complete task performance was minimal, indicating robustness across different complexity levels.
  • IAM Dataset: Achieved a new state-of-the-art F1-score of 85.7%, an improvement of 5.6% over a previous two-stage NER approach and 3.1% over the baseline transformer.
  • Qualitative Improvements: The model demonstrated better context understanding, successfully recognizing NE tags that were overlooked by baseline transformers. Examples showed improved handling of historical handwriting.

Limitations and Future Work

  • Overfitting with Full Attention: The paper notes that incorporating GCNs into a fully-attentive encoder (instead of the sparse SGTE) led to a drop in performance, suggesting potential overfitting. This highlights the importance of the sparse graph approach.
  • Minor Errors: Despite achieving state-of-the-art, the model occasionally makes errors, such as annotating words with uppercase characters that are not true named entities.
  • Generalizability: The authors suggest that the combination of Sparse Graph Transformer Encoder and Cross-GCN-based Decoder is a widely applicable approach that could be extended to other sequence-to-sequence applications and domains, implying future work could explore these extensions.

Summary updated

BibTeX
@inproceedings{khanfir2024graph,
  author = {Yessine Khanfir and Marwa Dhiaf and Emna Ghodhbani and Ahmed Cheikh Rouhou and Yousri Kessentini},
  title = {{Graph Neural Networks for End-to-End Information Extraction From Handwritten Documents}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure