Word prediction in computational historical linguistics

Peter Dekker, Willem H. ZuidemaJ. Lang. Model 2020 · DOI · OpenReview

TL;DR

This paper introduces "word prediction" as a novel machine learning task for computational historical linguistics, aiming to bridge the gap between phenotypic (surface form) and genotypic (historical sound changes) methods. By training models to predict a word's form in one language given its cognate in another, the authors demonstrate its utility for phylogenetic tree reconstruction, sound correspondence identification, and cognate detection, offering a more automated and data-driven approach to understanding language evolution.

What Problem Does This Paper Address?

Historical linguistics seeks to understand how languages are related and how they have evolved. Traditional methods, like the comparative method, rely on identifying cognates (words with a common ancestor) and regular sound correspondences to reconstruct language families and their histories. However, these manual methods are labor-intensive and scale poorly with the vast amounts of linguistic data now available. Computational approaches have emerged to automate these tasks, but often struggle to balance the need for genotypic information (sound changes) with the practicality of using readily available phenotypic data (word surface forms). This paper addresses the challenge of developing computational methods that can effectively leverage large datasets to reconstruct language ancestry, staying true to the core principles of the comparative method while embracing modern machine learning.

Methods

The core of this paper is the proposal and evaluation of word prediction as an intermediate task. The authors frame language reconstruction as a machine learning problem where a model learns sound correspondences between two languages by being trained on pairs of cognate words. This learned model can then predict the form of an unseen word in a target language given its cognate in a source language.

Key methodological components include:

  • Data Representation: Investigating three encoding schemes for phonetic data:
  • One-hot encoding: Simple binary representation for each phoneme.
  • Phonetic encoding: Using vectors of phonetic features (e.g., voiced, labial) to capture phoneme properties.
  • Embedding encoding: Learning vector representations of phonemes based on their context within words, inspired by word embeddings in NLP.
  • Machine Learning Models: Evaluating two sequence-to-sequence models:
  • Structured Perceptron: A simpler probabilistic sequence model.
  • RNN Encoder-Decoder: A more complex deep neural network architecture capable of handling sequential data.
  • Applications: Demonstrating the utility of word prediction by applying its outputs to:
  • Phylogenetic Tree Reconstruction: Using prediction distances between language pairs as a measure of ancestral relatedness to build evolutionary trees.
  • Sound Correspondence Identification: Analyzing the substitutions made by the prediction model to infer sound changes.
  • Cognate Detection: Clustering words based on prediction distances to identify potential cognates.

Datasets

The primary dataset used is the NorthEuraLex dataset, which contains phonetic word forms for 1016 concepts across 107 languages in Northern Eurasia. This dataset is noted for its size, exceeding typical datasets used in historical linguistics. For evaluation and comparison, the authors also refer to:

  • IELex dataset: Used for cognate judgments.
  • LexStat: Used for automatic cognate judgments when IELex data is unavailable.

The authors also mention using datasets from previous work for specific experiments, such as Romance language datasets (Ciobanu and Dinu 2018; Meloni et al. 2019) and Slavic datasets (Cathcart and Wandl 2020).

Key Insights and Ideas

  • Word Prediction as a Bridge: The central idea is that word prediction, by learning from surface forms but implicitly capturing sound changes, acts as a powerful intermediate step that bridges the gap between phenotypic and genotypic methods in historical linguistics.
  • Embedding Encoding's Potential: The embedding encoding, which learns phoneme representations from context, shows promise in capturing phonetic relationships, often grouping phonemes by their place of articulation, which is relevant for sound change.
  • Structured Perceptron's Efficacy: Despite its simplicity, the structured perceptron model often outperforms the more complex RNN encoder-decoder for the word prediction task, suggesting that simpler models can be highly effective for this specific problem.
  • Prediction Error as a Distance Metric: The prediction error (normalized edit distance) can serve as a useful, albeit imperfect, proxy for language distance, enabling applications like phylogenetic tree reconstruction.
  • Automating Cognate Judgments: Word prediction can be used to automate the identification of cognates, a crucial but labor-intensive step in traditional historical linguistics.

Key Results

  • Word Prediction Performance: The structured perceptron generally performed better than the RNN encoder-decoder across different encodings and language families (Slavic and Germanic).
  • Encoding Performance: Embedding encoding tended to perform best in most test cases for word prediction, though one-hot encoding was competitive for the Germanic family.
  • Phylogenetic Tree Reconstruction: The structured perceptron model, using embedding encoding and neighbor-joining clustering, produced phylogenetic trees for the Slavic language family with a low generalized Quartet distance (0.047619) to the reference Glottolog tree, indicating good performance.
  • Sound Correspondence Identification: The analysis of substitutions in the structured perceptron model revealed frequent sound correspondences that were also present between source and target words, suggesting the model learned meaningful linguistic patterns.
  • Cognate Detection: For the Germanic language family, the structured perceptron with MCL clustering achieved the highest B-Cubed F scores (0.932077), outperforming baselines. For the Slavic family, baselines performed slightly better, possibly due to closer language relatedness.

Limitations and Future Work

  • Prediction Error as Distance: The authors acknowledge that prediction error is not a perfect measure of language distance. It can be influenced by model performance and may not be informative for non-cognate word pairs. They suggest that ideally, a distance metric would be independent of model performance and account for both regular and irregular sound changes.
  • Data Representation Nuances: While embedding encoding showed promise, further research is needed to find optimal model architectures, parameter settings, and data encodings for maximizing word prediction performance.
  • Exploratory Nature: The paper is described as exploratory, aiming to understand the conditions under which the prediction paradigm can be used and to identify potential applications.
  • Future Research Directions: The authors suggest that future work should continue to explore prediction methods in historical linguistics, potentially modeling language evolution as a latent variable problem and identifying latent variables representing phonological processes of change. They also highlight the potential for applying these techniques to language families with ample data but unclear histories.

Summary updated

Abstract

In this paper, we investigate how the prediction paradigm from machine learning and Natural Language Processing (NLP) can be put to use in computational historical linguistics. We propose word prediction as an intermediate task, where the forms of unseen words in some target language are predicted from the forms of the corresponding words in a source language. Word prediction allows us to develop algorithms for phylogenetic tree reconstruction, sound correspondence identification and cognate detection, in ways close to attested methods for linguistic reconstruction. We will discuss different factors, such as data representation and the choice of machine learning model, that have to be taken into account when applying prediction methods in historical linguistics. We present our own implementations and evaluate them on different tasks in historical linguistics.
BibTeX
@article{dekker2020word,
  author = {Peter Dekker and Willem H. Zuidema},
  title = {{Word prediction in computational historical linguistics}},
  journal = {J. Lang. Model.},
  year = {2020},
  doi = {10.15398/jlm.v8i2.268},
}
Figure