Introduction
Building effective Natural Language Processing (NLP) tools requires vast amounts of clean, standardized data. For Indigenous languages like Plains Cree (nêhiyawêwin), this data is often fragmented or non-existent in digital formats. This research presents a new, processed corpus for Plains Cree, designed to bridge the digital divide and support the development of essential language technologies like predictive text and translation tools.
The Challenge of Polysynthetic Languages
Plains Cree is a polysynthetic and agglutinative language. Unlike English, where sentences are built from distinct words, a single Cree word can function as an entire sentence by combining multiple meaningful units, or morphemes. For example, the word kimîciso translates to "you all eat." This complexity makes standard word-level NLP models less effective, as the vocabulary size can be exponentially larger than in analytic languages.
Methodology: Rescue and Refine
The researchers employed a multi-stage "rescue and refine" workflow to curate the corpus:
- Extraction: Sourcing text from diverse origins, including educational materials, social media (Twitter/Facebook), and recordings of Elders.
- Standardization: Converting various writing systems—Syllabics and different Standard Roman Orthography (SRO) styles—into a consistent format using Python-based normalization scripts.
- Linguistic Rules: Implementing specific rules, such as the "h" joiner rule to handle vowel clashes during morpheme combination, ensuring a natural transition in text.
- Alignment: Manually and automatically aligning Cree sentences with English translations to create parallel data for machine learning.
Dataset Composition
The resulting corpus comprises 49,038 words and 1,026 lines of parallel English-nêhiyawêwin text. The sources vary significantly, reflecting both historical and modern usage, from Bible hymns to children's stories.
Ethical Considerations and the "Bible Bias"
A significant portion of the available data comes from religious texts. While technically valuable for training translation models, the authors acknowledge an ethical tension: these texts are artifacts of colonial history and may contain culturally insensitive content. To mitigate this, the project prioritized community involvement, ensuring the data and resulting tools—such as lesson plans and interactive worksheets—directly serve the needs of Cree speakers and educators.
Results and Future Directions
Initial tests using n-gram models for predictive text at the morpheme level showed promising results, with a 5-gram model achieving a perplexity of 133.12. Beyond technical metrics, the corpus is already being used in classrooms at the University of Alberta to create interactive learning materials. Future work will focus on expanding dialectal coverage, improving support for Syllabics, and moving toward more sophisticated neural network architectures as the dataset grows.
Summary updated