TL;DR
This paper investigates how different annotation strategies affect the performance of a Document Attention Network (DAN) for extracting information from handwritten Uruguayan birth certificates. They found that "normalized" annotations, which standardize fields like dates and places, work best for those fields, while "diplomatic" annotations, which preserve original text verbatim, are superior for names and surnames. This has implications for improving the efficiency and accuracy of digitizing historical documents.
What Problem Does This Paper Address?
Digitizing historical documents, particularly those with handwritten text like birth certificates, is a crucial but challenging task. These documents are essential for establishing legal identity, but many are still paper-based and lack digital counterparts. While automated information extraction (IE) systems can help, their performance heavily relies on the quality and format of the training data. This paper addresses the problem of how to best annotate handwritten documents for IE, specifically exploring two annotation strategies: "normalized" (standardized fields) and "diplomatic" (verbatim transcription). The goal is to understand which strategy is more effective for different types of information on birth certificates and how well a pre-trained model (DAN) can adapt to this specific task with limited data.
Methods
The researchers employed a pre-trained Document Attention Network (DAN) [11], an end-to-end architecture for handwritten text recognition and information extraction. DAN's key advantage is that it doesn't require manual segmentation of fields within an image, making it suitable for adapting to new document types with minimal annotation effort.
The core of their methodology involved fine-tuning DAN on a dataset of Uruguayan birth certificates using two distinct annotation strategies:
- Normalized Annotation: This approach uses existing annotations from the Civil Registry Office, where fields like dates and places are standardized.
- Diplomatic Annotation: This strategy involves a verbatim transcription of the text, preserving original casing, accents, and abbreviations as they appear in the handwritten document.
They then compared the performance of DAN fine-tuned with each annotation strategy on various fields within the birth certificates.
Datasets
The study utilized a dataset of 201 scanned Uruguayan birth certificates from four different years (2008, 2012, 2014, and 2016), handwritten by over 12 different writers. The original dataset contained transcriptions for seven key fields: "document number," "enrollee's full name," "year of enrollment," "jurisdiction," "1st parent's full name," "date of birth," and "2nd parent's full name." An additional field, "department," was added with no annotation cost.
From this original dataset, two new annotation sets were created:
- Normalized Annotation: Derived from the existing CSV files, with dates further processed into two fields ("date field 1" for the day in letters, and "date field 2" for the month in letters followed by the year).
- Diplomatic Annotation: A new set created by manually transcribing the documents verbatim, respecting accents, casing, and full names without abbreviations. This process took approximately 10 hours of manual work.
The dataset was split into training, validation, and testing sets (80%, 10%, 10% respectively).
Key Insights and Ideas
- Annotation Strategy Matters: The choice of annotation strategy significantly impacts the performance of information extraction models, especially when dealing with diverse data types within a single document.
- Standardization vs. Verbatim: For fields that can be standardized (like dates and places), normalized annotations lead to better results. This is likely because the model learns a consistent, predictable format.
- Names Require Nuance: For fields containing names and surnames, which are inherently less standardized and can include abbreviations, diplomatic annotations are crucial. Preserving the original text allows the model to capture these variations more accurately.
- DAN's Adaptability: The study demonstrates that DAN, even when pre-trained on a different language and document type (French letters), can be effectively fine-tuned for a new task (Spanish birth certificates) with a relatively small amount of data.
- Cost-Benefit of Annotation: While diplomatic annotation yields better results for names, it is more time-consuming. Normalized annotation is less costly and produces a directly usable formatted output, which is often preferred for database integration.
Key Results
The study found that Diplomatic DAN generally performed slightly better overall than Normalized DAN. However, a more detailed analysis revealed a clear pattern:
- Normalized DAN outperformed Diplomatic DAN in fields that do not contain names or last names (e.g., "year of enrollment," "jurisdiction," "department," "date field 1," and "date field 2"). For these fields, the normalized annotation approach was more cost-effective and produced a directly usable output.
- Diplomatic DAN significantly outperformed Normalized DAN in fields containing names and last names ("enrollee's full name," "1st parent's name," and "2nd parent's name"). For instance, the improvement in the "2nd parent's name" field was almost 3 times better when evaluated ignoring cases and accents. This highlights the importance of preserving original text for these types of fields.
Specifically, Table 2 shows that for "enrollee's full name," Diplomatic DAN had a CER of 2.94 (normalized) vs. 2.25 (normalized) for Normalized DAN, and a WER of 1.73 (normalized) vs. 9.64 (normalized). For "2nd parent's name," Diplomatic DAN achieved a CER of 5.70 (normalized) and WER of 28.36 (normalized), while Normalized DAN had a CER of 15.81 (normalized) and WER of 35.19 (normalized).
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Hybrid Annotation: They propose exploring a "hybrid" annotation strategy that combines normalized annotations for standardizable fields with diplomatic annotations for names and surnames. This could potentially offer the best of both worlds.
- Data Efficiency: Further research is needed to determine the minimum number of documents required to fine-tune DAN effectively without sacrificing accuracy.
- Generalization: Investigating DAN's generalization capabilities across different document layouts is a key future direction.
- Annotation Order: Testing the impact of different annotation orders for documents with varying margin placements (left vs. right) could improve accuracy.
- Document Redundancy: Exploiting inherent redundancy in documents (e.g., names appearing in both the margin and body) could enhance accuracy.
- Writer-Specific Models: Exploring whether training models on specific writer styles can lead to improved transcription accuracy is another promising avenue.
Summary updated