TL;DR
This paper details the creation of an open-source, linguistically annotated corpus from the 16th-century Florentine Codex, a monumental work in Nahuatl. By developing novel processing pipelines for retokenization and orthographic normalization, the authors aim to make this invaluable historical text searchable and analyzable for humanities scholars and computational linguists alike.
What Problem Does This Paper Address?
The Florentine Codex, a vast encyclopedic work compiled in the early colonial period, is a treasure trove of information about Nahua culture, history, and language. However, its historical nature presents significant challenges for computational analysis. These include:
- Orthographic Inconsistencies: The text was written in the early days of alphabetic Nahuatl, leading to numerous variations in spelling and word representation.
- Tokenization Ambiguities: Words are frequently split across lines due to manuscript constraints, making it difficult to identify word boundaries.
- Morphological Complexity: Nahuatl is a highly agglutinative and polysynthetic language, meaning words can be very long and complex, making it hard to search for base forms (lemmas).
- Copyright Restrictions: Existing digitized versions are often under copyright, limiting their use for annotation and derivative works.
These issues make traditional keyword searching and computational linguistic analysis extremely difficult, hindering scholarly research.
Methods
The authors tackle these challenges by building a robust processing pipeline for the Florentine Codex, focusing on creating an open and extensible machine-readable corpus. Key methods include:
- Retokenization: This crucial step addresses the issue of words split across lines and inconsistent word boundaries. They developed a rule-based system, employing a left-to-right longest-match algorithm with approximately 4,000 rules. This process aims to align word boundaries with canonical Nahuatl words.
- Orthographic Normalization: To handle spelling variations, they use the ACK (Andrews, Campbell, Karttunen) standard for Nahuatl and contemporary orthography for Spanish words. This involves creating a hand-curated dictionary mapping original forms to normalized ones, bootstrapped by a finite-state transducer model.
- Part-of-Speech (POS) Tagging: They employ a multi-step process using a lexicon, a morphological analyzer, and a set of regex-based guessing rules.
- Morphological Analysis: This involves producing a lemma (base form) and morphosyntactic tags for each word. They utilize the Helsinki Finite-State Toolkit (HFST) for this, implementing morphotactics and lexical analysis.
- Automated Processing Experiments: The authors experiment with Seq2Seq models with attention for both retokenization and orthographic normalization, treating them as character-based machine translation problems. They use books 1 and 8 for training and book 5 for evaluation.
Datasets
- Florentine Codex: The primary dataset is a typed transcription of the Nahuatl portion of the Florentine Codex, sourced from the open-access repository Zenodo.
- Training Data: For automated processing experiments, books 1 and 8 of the Florentine Codex were used for training.
- Evaluation Data: Book 5 of the Florentine Codex was used for evaluating the automated models.
Key Insights and Ideas
- The Power of Open Access: Creating an open-source, annotated corpus is vital for democratizing access to historical texts and fostering interdisciplinary research.
- Rule-Based Systems as a Strong Baseline: For tasks like retokenization, a carefully crafted rule-based system can achieve high precision and recall, especially when combined with manual auditing.
- Machine Translation for Text Normalization: Treating orthographic normalization as a character-based machine translation problem shows promising results, even with limited training data.
- The Importance of Intermediate Steps: Separating tasks like retokenization and normalization allows for more granular analysis and potential future research on how these steps interact.
- Nahuatl's Linguistic Richness: The paper highlights the complexities of Nahuatl, particularly its incorporation feature, which requires special attention in linguistic annotation frameworks.
Key Results
- Manual Processing Progress: The authors have nearly completed retokenization, orthographic normalization, lemmatization, POS tagging, and morphological analysis for 3 out of the 12 books of the Florentine Codex.
- Orthographic Normalization Accuracy: The orthographic normalization model correctly normalizes 87% of words in the held-out book.
- Retokenization Model Performance: Experiments show favorable results, though not yet at a level to completely automate the process. The sequence labeling approach for retokenization achieved a word error rate of 23.9% and a character error rate of 2.8%.
- Automated Normalization: The NMT + Dictionary approach for orthographic normalization achieved a CER of 1.7% and WER of 8.3%.
- Manual Guessing Accuracy: Manually checked guesses for POS tagging were highly accurate, with 198 out of 200 being correct.
Limitations and Future Work
- Limited Training Data: The relatively small volume of processed data for training automated models is a significant limitation, leading to issues like "hallucinations" in the retokenization model.
- Automation vs. Manual Auditing: While automation is a goal, manual auditing of the entire annotated corpus is still required to ensure high quality.
- Future Work:
- Continue annotation for the remaining books.
- Automate more text normalization tasks.
- Expand the lexica and enhance the morphological analyzer.
- Add dependency syntax annotations for quantitative analysis of Nahuatl syntax.
- Explore more powerful architectures and data augmentation techniques for retokenization.
- Translate the corpus into contemporary Nahuatl variants to increase accessibility.
Summary updated