TL;DR
This paper tackles the challenge of recognizing historical handwriting when labeled data is scarce. They propose a novel character-based decoder that leverages transfer learning from various sources, including historical documents and modern text, to infer words from letter-ngram representations. Their approach achieves impressive results, demonstrating the effectiveness of carefully selected training data for historical handwriting recognition.
What Problem Does This Paper Address?
Digitizing historical documents is a crucial step in preserving cultural heritage, but making this information accessible for research requires more than just images. Extracting text from historical handwriting (Handwriting Recognition or HWR) is a significant challenge due to variations in writing styles, languages, and document degradation. A major hurdle in developing robust HWR systems is the scarcity of labeled data, especially for specific historical collections where creating ground truth is time-consuming and expensive. This paper specifically addresses the problem of building an effective HWR system for historical Italian Comedy registers, which suffer from a lack of ground truth and present complexities like multilingualism and evolving writing styles.
Methods
The authors propose an encoder-decoder architecture for HWR, inspired by image captioning models. The system consists of two main components:
- Image Encoder: This part takes a word image and converts it into a vector of letter-ngrams. It uses a fully convolutional network (FCN) followed by fully connected layers.
- Character-Based Decoder: This component takes the letter-ngram vector and generates the corresponding word character by character. It employs a minimalist architecture with a fully connected layer, a Gated Recurrent Unit (GRU) layer for temporal processing, and a softmax activation function.
A key innovation is the use of letter-ngrams (combinations of 1 to 3 consecutive characters) as an intermediate representation. This pivot allows for transfer learning across different datasets, even those with varying languages and writing styles, as long as they share the same alphabet. The decoder is trained to infer the order of characters solely from these n-gram frequencies, without relying on explicit word order information. They also explore the impact of using a dictionary during decoding, finding that it can sometimes degrade performance.
Datasets
The paper utilizes several datasets for training and evaluation:
- Target Data:
- Italian Comedy Registers (ICR): A newly collected and validated dataset of 971 annotated title lines from financial records of the Italian Comedy (1716-1791). This dataset serves as the primary target for evaluation.
- Training/Transfer Datasets:
- Google Italian Comedy (GIC): A new dataset created from digitized 18th-century books related to the Italian Comedy, aiming to provide a similar closed vocabulary and writing style to the ICR.
- RIMES (RM): A French database for evaluating handwritten text recognition systems.
- George Washington (GW): An English database from the George Washington Papers.
- Los Esposalles (ESP): A Spanish database from 15th and 17th-century marriage license books.
- Wikipedia Fr data (Wiki): A large vocabulary dataset from Wikipedia, used to assess the impact of modern, general-purpose resources.
Key Insights and Ideas
- Letter-Ngrams as a Transferable Pivot: Using letter-ngrams as the interface between the encoder and decoder proves to be a robust strategy for transfer learning. This representation is transferable across languages and historical periods, provided the character set is consistent.
- Historical Data Superiority: For historical handwriting recognition, using other historical resources (like GIC) as training data is more effective than relying solely on modern, general-purpose resources like Wikipedia. This suggests that historical spelling variations are easier for the decoder to learn from.
- Dictionary's Double-Edged Sword: While dictionaries are common in HWR, their use in this transfer learning context can sometimes hinder performance. This is attributed to the dictionary's limited coverage of historical spelling variations and the potential for it to mislead the decoder when encountering slightly different word forms.
- Minimalist Decoder Effectiveness: A simple, character-based decoder with a limited number of layers (GRU) can achieve strong results when trained with appropriate data, demonstrating that complex architectures are not always necessary.
Key Results
The paper reports significant achievements:
- High Recognition Rates: The system achieves a 97.42% Character Recognition Rate (CRR) and an 86.57% Word Recognition Rate (WRR) on the Italian Comedy data.
- Effective Transfer Learning: Even with a lexical coverage of only 67% between the training data (GIC) and the target data (ICR), the transfer learning approach yields excellent results.
- Outperforming Baselines: The letter-ngram approach significantly outperforms a baseline using only unigrams, showing a substantial increase in WRR.
- Dictionary Impact: Using a dictionary generally degrades performance, especially when lexical coverage is low. For instance, with RM, WRR drops from 41.18% to 25.24% when a dictionary is used.
Limitations and Future Work
The authors acknowledge several limitations and propose future directions:
- Lexical Coverage: While results are good, the lexical coverage between training and test sets can be low, especially when using modern datasets like Wikipedia for historical data.
- Specific Word Errors: The decoder sometimes struggles with words containing repeated characters or specific abbreviations.
- Punctuation and Spaces: The current system focuses on word-level recognition and does not handle punctuation marks or spaces.
- Noisy Input: The current decoder receives clean letter-ngram vectors. Future work aims to develop a more robust handwriting recognition system with an encoder-decoder model where the decoder input might be noisy (i.e., directly from the encoder without perfect n-gram extraction).
- Sentence-Level Decoding: Extending the decoder to handle sentence-level recognition, including spaces, is a key future goal.
- Encoder Component Improvement: Further research will focus on improving the encoder component to better handle errors and extract more informative features.
Summary updated