TL;DR
This paper tackles the challenge of preserving linguistic diversity by digitizing historical documents of Peruvian indigenous languages using OCR. They introduce a novel alignment correction module and evaluate several post-OCR correction methods, demonstrating that a sequence-to-sequence model is most effective for these low-resource languages.
What Problem Does This Paper Address?
The rapid advancement of Natural Language Processing (NLP) technologies has largely benefited resource-rich languages, leaving thousands of minority languages behind. This digital divide exacerbates the loss of cultural and linguistic heritage, especially when historical resources are only available in difficult-to-access formats like old, scanned books. This paper focuses on Peruvian indigenous languages, which often lack digital presence and have their valuable historical texts locked away in non-digital forms, hindering their preservation and the development of language technologies for their speakers.
Methods
The core of this work involves a two-pronged approach: digitizing historical documents using Optical Character Recognition (OCR) and then correcting the resulting errors.
- OCR Systems: They employed two established OCR systems, Google Vision and Tesseract, which, despite not having direct support for the target languages, can process the Latin script they use.
- Preprocessing: Before correction, they performed noise removal (e.g., page numbers), attribute removal (e.g., boxes around text), and image enhancement to improve OCR accuracy.
- Alignment Correction: A significant novel contribution is a module developed to automatically align OCR transcriptions. This was crucial for Google Vision, which struggled with text order in multi-column layouts, improving Character Error Rate (CER) by approximately 9% and Word Error Rate (WER) by 12%. Tesseract, however, did not require this step due to its better text block segmentation.
- Post-OCR Correction Models: They evaluated five methods:
- SingleSource: A sequence-to-sequence model designed for low-resource scenarios.
- Denorm: A spell checker trained on the specific languages.
- Ensemble: A more comprehensive spell checker addressing various error types.
- SingleSource+Denorm: A cascaded approach.
- SingleSource+Ensemble: Another cascaded approach.
Datasets
The primary dataset consists of 454 scanned pages from 89 books in four Peruvian indigenous languages: Asháninka, Shipibo-Konibo, Yanesha, and Yine. This dataset comprises 31,423 tokens. This is a newly created benchmark for OCR and post-OCR correction in these specific languages.
Key Insights and Ideas
- The "digital divide" is a real threat to linguistic diversity: The paper powerfully illustrates how the lack of digital resources for minority languages leads to a loss of cultural heritage.
- OCR is a viable first step for digitizing historical texts: Even with imperfect results, OCR can unlock information from old documents that would otherwise be inaccessible.
- Alignment is a critical, often overlooked, OCR challenge: For systems like Google Vision, correctly ordering text, especially in complex layouts, is as important as character recognition itself.
- Specialized models are key for low-resource languages: General-purpose spell checkers struggle with the unique characteristics of historical texts from indigenous languages, which may not adhere to standardized alphabets or norms.
- Character-level errors, especially diacritics, are a major hurdle: The misrecognition of characters with diacritics was a dominant error type, highlighting the need for models that can handle these specific linguistic features.
Key Results
- The SingleSource model was the most effective post-OCR correction method, significantly improving recognition of characters with diacritics (up to 65% for Yanesha) and reducing errors in word boundary identification.
- Google Vision OCR showed higher accuracy in character recognition, while Tesseract excelled in maintaining text order.
- The developed alignment module reduced CER by ~9% and WER by ~12% for Google Vision OCR.
- The SingleSource model achieved a 99% accuracy in identifying characters with diacritics for Shipibo-Konibo and Yanesha.
- Despite improvements, character deletion was a primary error type for the SingleSource model, with correction ratios varying by language and OCR system.
Limitations and Future Work
- Limitations:
- The SingleSource model sometimes introduced character deletions.
- The effectiveness of standardization-based spell checkers (Denorm, Ensemble) was limited due to the historical texts not adhering to official alphabets.
- Uneven distribution of low-frequency characters in the dataset partitioning could impact model performance.
- Future Work:
- Optimize hyperparameters for the SingleSource model.
- Implement a multi-source model that leverages Spanish translations (available for 89% of the books) to further improve correction accuracy.
Summary updated