Investigating variation in written forms of Nahuatl using character-based language models

Robert Pugh, Francis TyersNAACL 2021

TL;DR

This paper explores how character-based language models can effectively capture linguistic variation in written Nahuatl, a language with many dialects. By training simple LSTM models on Bible translations, the authors demonstrate their ability to identify variants, group them by similarity, and even predict mutual intelligibility between them. This work highlights the power of character-level modeling for low-resource languages and offers a computational approach to understanding linguistic diversity.

What Problem Does This Paper Address?

The study of linguistic diversity, particularly in languages with many variants like Nahuatl, presents a significant challenge for natural language processing (NLP). Marginalized and endangered languages often lack the extensive resources (like large annotated datasets) that are common for more widely spoken languages. This scarcity makes it difficult to develop effective language technologies. Furthermore, understanding the relationships between different variants – how similar they are, how mutually intelligible they are, and how they are related historically – is crucial for language revitalization efforts and for building accurate NLP tools. This paper tackles this by investigating whether simple, character-based language models, which require minimal preprocessing, can effectively capture and quantify the similarities and differences between written Nahuatl variants.

Methods

The core of this research relies on character-based language modeling using a unidirectional LSTM (Long Short-Term Memory) recurrent neural network. This approach treats each character as a distinct token, learning to predict the next character in a sequence. This is a standard technique in NLP, but its application here to a low-resource language with significant orthographic variation is notable.

The authors trained these models on parallel Bible translations for various Nahuatl variants. They then used the models to perform three main tasks:

  1. Variant Identification: Predicting which variant a given text belongs to by selecting the model that assigns the lowest perplexity (a measure of how "surprised" the model is by the text).
  2. Sub-classification and Genetic Similarity: Clustering variants based on their cross-variant perplexity. This means measuring how well a model trained on one variant predicts texts from another variant. High perplexity suggests low similarity.
  3. Mutual Intelligibility: Comparing the cross-variant perplexity scores with existing linguistic studies on mutual intelligibility between Nahuatl speakers.

A key methodological step was orthographic normalization to mitigate variations in spelling across different translations, ensuring that the models focused on linguistic rather than orthographic differences.

Datasets

The primary dataset used in this study consists of Bible translations for 10 different Nahuatl variants, sourced from scriptureearth.org. To ensure comparability, only verses present in all 10 variants were used, resulting in a dataset of 7,363 verses per variant.

The study also leverages existing linguistic data for comparison, specifically:

  • Ethnologue (Eberhard et al., 2021) for mapping ISO-639 codes to towns and municipalities.
  • Egland and Bartholomew (1978), a seminal study on mutual intelligibility between 58 Nahuatl communities, which provided percentage scores for intelligibility between community pairs.

Key Insights and Ideas

  • Character models are surprisingly powerful for linguistic variation: Even simple character-level LSTMs can capture significant linguistic information, including structural phonological and grammatical differences, without needing complex linguistic features or tokenization.
  • Orthographic normalization is crucial: The authors' decision to normalize orthography was essential for disentangling linguistic variation from spelling differences, allowing the models to focus on the underlying language.
  • Cross-variant perplexity as a proxy for similarity: The study demonstrates that the perplexity a model trained on variant A assigns to text from variant B can serve as a quantitative measure of linguistic similarity between A and B.
  • Mutual intelligibility is correlated with linguistic similarity: The research found a moderate negative correlation between cross-variant perplexity and reported mutual intelligibility, suggesting that character language models can offer insights into how well speakers of different variants might understand each other.
  • Huasteca variants are highly similar: The Huasteca variants (nhw, nhe, nch) showed the lowest cross-variant perplexity, indicating a high degree of similarity, which aligns with linguistic descriptions.

Key Results

  • Variant Identification: Achieved near-perfect accuracy (0.99) in identifying the correct variant for a given text. The few errors occurred between closely related Huasteca variants.
  • Sub-classification: Hierarchical clustering based on cross-variant perplexity showed a general correspondence with established linguistic groupings of Nahuatl variants (e.g., Central-Periphery, Huasteca).
  • Mutual Intelligibility: A moderate negative correlation (r(19) = -0.734, p = .0002) was found between cross-variant perplexity and reported mutual intelligibility percentages from Egland and Bartholomew (1978). This relationship was particularly strong for the most similar variants.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Domain Specificity: The high performance in variant identification might be due to the restricted domain (Bible translations) and the fact that all texts for a given variant were likely produced by the same translator(s). This could lead to models capturing author-specific style rather than purely linguistic variant features.
  • Limited Representation of Some Groups: The cross-variant clustering did not clearly distinguish the "Western Periphery" and "Eastern Periphery" groups, possibly due to the limited number of variants from these groups in the dataset.
  • Correlation Strength: While moderate, the correlation between perplexity and mutual intelligibility is not perfect, especially for less similar variants.

Future work suggested includes:

  • Exploring other tokenization schemes, such as unsupervised subword tokenization and morphological segmentation.
  • Investigating the inclusion of variants not present in the current dataset (e.g., Isthmus Mecayapan Nahuatl - nhx).
  • Applying these models to non-parallel or comparable text to assess their robustness beyond the controlled Bible corpus.

Summary updated

BibTeX
@inproceedings{pugh2021investigating,
  author = {Robert Pugh and Francis Tyers},
  title = {{Investigating variation in written forms of Nahuatl using character-based language models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure