TL;DR
This paper tackles the critical lack of digital NLP resources for Ancash Quechua, a variety of the Central Quechua language. The authors successfully digitize a key Ancash Quechua-Spanish dictionary using OCR, adapting existing tools and developing a new, specialized OCR model. This work lays the groundwork for future NLP development for this under-resourced language.
What Problem Does This Paper Address?
Natural Language Processing (NLP) has seen remarkable advancements, but these benefits are not evenly distributed across all languages. Many indigenous languages, including various Quechua dialects, suffer from a severe deficit of digital resources like corpora and lexicons. While Southern Quechua has seen some development, Central Quechua varieties, such as Ancash Quechua, are largely neglected. This lack of resources hinders the development of NLP tools that could support language preservation, education, and broader accessibility. Specifically, digitizing existing, often paper-based, lexical resources is a crucial first step, but it's challenging due to the unique characteristics of these languages and the limitations of general-purpose OCR tools.
Methods
The authors employed a multi-stage approach to create a digital Ancash Quechua lexical resource:
- OCR Software Comparison: They began by evaluating three OCR software packages (ABBYY Finereader, Tesseract, and GoogleDocs OCR) on a sample of the Ancash Quechua dictionary. This step aimed to identify the most promising off-the-shelf solution.
- OCR Model Adaptation: Based on the comparison, they chose Tesseract due to its open-source nature and potential for customization. They then retrained Tesseract's existing Quechua model to specifically adapt it to the Ancash variety and the unique typography of the source document (which mimicked old typewriters). This involved creating a training corpus from manually corrected pages of the dictionary.
- Digitization and Lexical Extraction: The adapted OCR model was used to digitize the entire dictionary. The extracted lexical information was then structured and stored in an XML-TEI format, a standard for encoding dictionaries.
- Preprocessing: Before OCR, scanned pages underwent preprocessing, including cropping and conversion to greyscale with increased contrast and high-definition PNG format.
Datasets
The primary dataset used in this work is a working draft of the Ancash Quechua to Spanish dictionary by Gary J. Parker (1975). This document, described as a list of Ancash lexemes with usage information, POS tags, Spanish translations, and cross-references, served as both the source for OCR and the basis for creating the training data.
For training the specialized Tesseract model, a corpus of 30 pages (approximately 5,676 words and 33,687 characters) was manually corrected.
Key Insights and Ideas
- Tesseract's Adaptability is Key: While GoogleDocs OCR showed the best raw performance initially, Tesseract's ability to be retrained and its open-source license made it the superior choice for creating a shareable, specialized model. This highlights the importance of customizable OCR for under-resourced languages.
- Typography Matters: The unique typography of the source document, resembling old typewriters, presented challenges for standard OCR models. Adapting the model to these specific characters and their representations was crucial for accurate recognition.
- Manual Correction is Still Essential: Despite advancements in OCR, manual correction remains a vital step, especially for specialized documents. The authors found that their trained model significantly eased this process, making errors more regular and easier to detect.
- XML-TEI for Interoperability: Using the XML-TEI format for the final lexical resource ensures that it can be easily used by various NLP tools and researchers, promoting wider adoption and utility.
Key Results
- The adapted Tesseract model achieved a Character Error Rate (CER) of 2.57% and a Word Error Rate (WER) of 8.19% on a test set of 50 randomly selected pages.
- The specialized model showed a significant improvement over the generic Quechua+Spanish model, with a 3% increase in character recognition accuracy and a 13% increase in word recognition accuracy.
- The manual correction time per page was estimated to be around 3 minutes and 40 seconds using the new model, indicating a substantial efficiency gain.
- The resulting XML-TEI lexicon contains 3,626 entries, making it the largest digital resource for Ancash Quechua available for NLP and lexicometry to date.
Limitations and Future Work
The authors acknowledge that the source document is a working draft and may contain inaccuracies. While their OCR model significantly improved recognition, some diacritics were still challenging, and the structure of the document was not perfectly preserved by all OCR tools.
Future work includes:
- Digitizing a second dictionary and a corpus with similar characteristics to further enrich Ancash Quechua resources.
- Expanding the use of the developed OCR model to other similar documents.
- Encouraging further digitization initiatives for under-resourced languages.
Summary updated