In search of isoglosses: continuous and discrete language embeddings in Slavic historical phonology

Chundra Cathcart, Florian WandlACL 2020

TL;DR

This paper explores how neural networks can learn historical sound changes in Slavic languages by using different types of language embeddings. They found that a "straight-through" discrete embedding approach achieved the best accuracy, while a "sigmoid" embedding better reflected traditional Slavic language groupings, suggesting a trade-off between predictive power and linguistic interpretability.

What Problem Does This Paper Address?

Historical linguistics often relies on identifying "isoglosses" – linguistic features that mark boundaries between languages – to reconstruct language families and their evolutionary paths. While the subgrouping of Slavic languages is generally well-established, there are debates about which features are most indicative of historical relationships. This paper tackles the challenge of using computational methods, specifically neural networks, to automatically learn these historical phonological patterns and assess their ability to capture genetic relationships between languages. The authors aim to see if neural models can not only predict sound changes but also learn language-level information that reflects established linguistic classifications, and to explore the interpretability of these learned representations.

Methods

The core of the paper's methodology is an LSTM Encoder-Decoder architecture. This model is trained to reconstruct Proto-Slavic etymons (ancestral forms) into their modern Slavic reflexes (descendant forms). A key innovation is the use of different types of language embeddings concatenated to the input at each time step. These embeddings are designed to represent the target language and are varied across experiments:

  • Dense: A standard continuous embedding.
  • Sigmoid: A continuous embedding constrained to the [0, 1] interval.
  • Straight-Through (ST): A discrete, binary embedding. This is achieved by using a "straight-through estimator" which allows gradients to flow through the discrete layer during backpropagation, despite the discrete nature of the output. This is a computationally efficient way to work with discrete latent variables in neural networks.

The models are trained on an etymological dictionary and evaluated using word error rate (WER) and phoneme error rate (PER). They also analyze the learned embeddings by constructing phylogenetic trees and examining error types.

Datasets

The primary dataset used is derived from a digitized Slavic etymological dictionary (Derksen 2007), containing Proto-Slavic etyma and their corresponding reflexes in 13 medieval and modern Slavic languages. For modern languages, IPA transcriptions were obtained from Wiktionary, with manual checks and corrections. Historical and phonetic descriptions from various sources were also consulted to ensure data quality. Languages with fewer than 100 forms were excluded.

Key Insights and Ideas

  • Discrete Embeddings Excel in Accuracy: The "straight-through" (ST) discrete language embeddings significantly outperformed continuous (Dense and Sigmoid) embeddings in terms of predictive accuracy (lower WER and PER). This suggests that a binary, discrete representation might be more effective for capturing the specific patterns of sound change in this task.
  • Sigmoid Embeddings Reflect Linguistic Grouping: Despite lower accuracy, the Sigmoid embeddings showed the strongest agreement with the traditional tripartite classification of Slavic languages (East, West, South) when analyzed via phylogenetic trees. This highlights a potential trade-off: models that are better at predicting might not necessarily learn representations that align with established linguistic theories.
  • Interpretability is Challenging but Promising: The ST model's discrete latent space, while not perfectly sparse, showed potential for learning meaningful information about sound change. However, individual dimensions didn't isolate specific phonological rules; rather, interactions between dimensions seemed to drive the model's behavior.
  • Data Quality Matters: The authors note that a significant portion of errors can be attributed to "target errors" – morphological mismatches or ambiguities in the source data – rather than solely model failures.

Key Results

  • Accuracy: The ST model achieved the lowest mean word error rate (WER) of 0.530 and phoneme error rate (PER) of 0.140. The Dense model had a WER of 0.535 and PER of 0.143, while the Sigmoid model had the highest WER of 0.559 and PER of 0.151.
  • Phylogenetic Agreement: The Sigmoid model's embeddings resulted in a phylogenetic tree with the lowest generalized quartet distance (0.247) compared to the reference Glottolog tree, indicating better alignment with traditional linguistic subgroupings. The Dense model had a distance of 0.322, and the ST model had the highest distance at 0.368.
  • Error Analysis: The ST and Sigmoid models produced more "other-language" and "unmotivated" errors compared to the Dense model, suggesting they might be more prone to cross-linguistic influence or generating novel, unattested patterns.

Limitations and Future Work

  • Sparsity of Embeddings: The learned discrete embeddings were not as sparse as ideal for direct interpretability, meaning individual dimensions didn't map cleanly to single phonological features.
  • Data Artifacts: A notable portion of errors stemmed from imperfections in the training data, such as morphological mismatches.
  • Future Directions:
    • Variational Approaches: The authors plan to explore more advanced variational methods to induce sparsity and learn more meaningful, interpretable representations.
  • Broader Data: Extending the analysis to include more language families and a wider range of linguistic phenomena.
  • Inter-annotator Reliability: Conducting more detailed qualitative error analyses with multiple annotators to ensure reliability.
  • Investigating Interactions: Further research into how interactions between latent dimensions influence model output.

Summary updated

BibTeX
@inproceedings{cathcart2020search,
  author = {Chundra Cathcart and Florian Wandl},
  title = {{In search of isoglosses: continuous and discrete language embeddings in Slavic historical phonology}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure