Understanding Native Language Identification for Brazilian Indigenous Languages

Paulo Cavalin, Pedro Domingues, Julio Nogima, Claudio PinhanezACL 2023

TL;DR

This paper tackles the challenge of identifying Brazilian Indigenous Languages (BILs) using machine learning, a crucial step for creating vital digital resources for these endangered languages. By training a language identification model on Bible texts and evaluating it on diverse, non-biblical data, the authors reveal both the potential and significant limitations of this approach, particularly concerning the influence of closely related languages.

What Problem Does This Paper Address?

Brazil is home to approximately 270 indigenous languages, many of which are endangered and severely understudied. The creation of AI tools like language translators, spell checkers, and next-word predictors is essential for revitalizing these cultures and supporting language learning. However, a major hurdle in developing these tools for understudied languages is the scarcity of readily available data in machine-readable formats. Native Language Identification (LangID) offers a promising avenue for automatically discovering and gathering such data from the web.

While previous LangID work has often relied on limited, domain-specific datasets (like the Bible), this paper investigates the practical performance of a Bible-trained LangID system on "out-of-domain" data. Specifically, it aims to understand how well a model trained on religious texts generalizes to everyday language and what challenges arise when applying it to real-world, diverse text collections.

Methods

The researchers implemented a language identification system using a linear Support Vector Machine (SVM) classifier. They employed a "bag-of-words" (BoW) feature extraction method, which represents text by the frequency of words it contains. For tokenization (breaking text into meaningful units), they used SentencePiece with a vocabulary size of 100,000 tokens. This approach was chosen after evaluating various configurations and classifiers, with the linear SVM and 100K tokens yielding the best performance on their test sets.

Datasets

The study utilized two primary types of datasets:

  • Bible-based training data: The authors created a new dataset by collecting approximately 1.5 million sentences from 51 Brazilian Indigenous Languages (BILs) found in Bible translations. This was combined with the existing WiLi2018 dataset, which contains 235 languages, to form the main training corpus.
  • Non-biblical evaluation datasets: To assess generalization, seven new datasets were created, comprising six different BILs. These datasets were sourced from various repositories, including PDF files and annotation efforts like Universal Dependencies Parsing (UDP). This "out-of-domain" data is crucial for understanding real-world performance.
  • Portuguese Wikipedia: A large dataset of approximately 13.5 million sentences from the Portuguese Wikipedia was used for an "in-the-wild" evaluation, simulating a practical application scenario.

Key Insights and Ideas

  • Bible data is a double-edged sword: While Bible texts provide a crucial, albeit limited, source for training LangID models for understudied languages, they may not fully represent the diversity of language use in everyday contexts.
  • Generalization varies by language: The Bible-trained LangID model performs reasonably well on languages with more established writing standards (e.g., Guarani Mbya, Kaigang) but shows a significant drop in accuracy for languages with less standardized orthography (e.g., Apurinã).
  • Closely related languages are a major challenge: When applied to large, real-world datasets like Wikipedia, the model struggles with distinguishing between similar indigenous languages, leading to a high rate of false positives. This suggests that a more comprehensive approach, including a wider range of related languages, is necessary.
  • Manual inspection is invaluable: Despite the limitations, manual inspection of misclassified sentences provided critical insights into the difficulties of LangID for BILs and highlighted the need for better handling of linguistic similarities.

Key Results

  • The Bible-based LangID model achieved an accuracy of 73.3% on a test set of BILs and 95.1% on non-BIL languages.
  • On out-of-domain, non-biblical datasets, accuracy varied significantly by language, ranging from a high of 91.7% (for myu) to a low of 37.3% (for apu). Languages with more established written forms generally performed better.
  • When applied to the Portuguese Wikipedia, the model identified only 0.03% of sentences as potentially belonging to one of the 51 BILs. After filtering for sentences with low confidence scores, only 129 sentences remained.
  • Manual inspection of these 129 sentences revealed a precision of only 7%, indicating a substantial problem with false positives.
  • A significant portion of misclassifications involved languages from the Tupi-Guarani family, highlighting the difficulty in distinguishing between closely related languages.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited scope of LangID methods: The study focused on one specific LangID approach, and exploring other methods could yield different results.
  • Limited non-BIL languages: The WiLi2018 dataset, while extensive, could be expanded with even more non-BIL languages to improve robustness.
  • Wikipedia data bias: Relying on Wikipedia limits the scope to text that is likely more formal and potentially biased towards Portuguese. A broader dataset like Common Crawl would offer a more realistic "in-the-wild" scenario.
  • Lack of native speaker validation: The crucial step of having native speakers review the results is still in its early stages, with engagement only beginning with one Mbya Guarani community.

Future work suggested includes:

  • Expanding the training set: Incorporating a much larger and more diverse set of languages, especially from the Americas, is paramount.
  • Orthography inspection: Collaborating with linguists and native speakers to understand and potentially standardize orthographic variations.
  • Broader in-the-wild data exploration: Investigating larger, more diverse datasets like Common Crawl and BrWac, and exploring PDF document mining for data.
  • Developing mechanisms for out-of-scope detection: Creating methods to better handle languages that are not explicitly included in the training data.

Summary updated

BibTeX
@inproceedings{cavalin2023understanding,
  author = {Paulo Cavalin and Pedro Domingues and Julio Nogima and Claudio Pinhanez},
  title = {{Understanding Native Language Identification for Brazilian Indigenous Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure