A Probabilistic Approach to Language Change

Alexandre Bouchard-Côté, Percy Liang, Thomas L. Griffiths, Dan KleinNeurIPS 2007 · OpenReview

Historical linguistics has traditionally relied on the "comparative method"—a manual, often subjective process where linguists compare words across languages to identify regular sound correspondences and infer ancestral "proto-languages." Previous computational attempts to automate this process often simplified the problem too much. For instance, glottochronology treats the presence or absence of a word in a language as a single bit of data, ignoring the rich information contained in the actual sounds (phonemes) of the words. While newer models incorporated phonemes, they struggled with "granularity": they either treated every sound change in every context as a unique event (leading to data sparsity) or were too broad to capture specific linguistic nuances.

This paper introduces a probabilistic framework for modeling how word sounds evolve across a family tree of languages. By using a log-linear model to represent phonological "rules" with linguistically-motivated features, the authors provide a way to automatically and accurately reconstruct ancient word forms (like Latin) from their modern descendants (like Spanish and Italian).

Methods: A Generative Model of Phonological Change

The authors propose a generative model of phonological change that bridges the gap between statistical robustness and linguistic theory. The framework is built on three key components:

  1. Phylogenetic Trees: Languages are arranged in a known tree structure. Word forms start at the root (the ancestral language) and undergo stochastic (random) edits as they move down the branches to become modern languages.
  2. Stochastic Edit Model: Changes along the branches are modeled as sequences of three operations: deletion, substitution, and insertion. Crucially, the probability of an edit depends on its context—specifically, the preceding and following phonemes.
  3. Log-Linear Rule Distribution: This is the paper's core innovation. Instead of using simple probability tables, the authors use a log-linear model to define the probability of a rule. This allows them to incorporate linguistic features (e.g., "is the phoneme a vowel?" or "is it at the end of a word?"). This "featurized" approach enables the model to learn general rules that apply across many sounds while still capturing highly specific contextual changes.
Overview of the generative model and stochastic edits.

For inference, the model employs a Monte Carlo Expectation-Maximization (EM) algorithm. The "E-step" uses Gibbs sampling and dynamic programming to estimate which edits likely occurred, while the "M-step" uses the L-BFGS optimization algorithm to update the weights of the linguistic features.

Datasets and Evaluation

The model was evaluated using a dataset of 582 cognate sets (groups of words with a common origin) from the Romance language family. The tree included Latin at the root and its descendants: French, Spanish, Italian, and Portuguese.

To test the model, the authors held out a portion of the Latin words and tasked the system with reconstructing them based entirely on the modern forms and the known tree structure. They also validated the learned "rules" against historical evidence, such as the Appendix Probi—a 3rd/4th-century list of common Latin misspellings that provides historical clues to how the language was actively changing.

Results: Reconstructing Ancient Words and Sound Laws

The featurized log-linear model demonstrated significant advantages over baseline approaches:

  • Reconstruction Accuracy: The model significantly outperformed the baseline, achieving a 14% improvement in edit distance over a "random neighbor" baseline. In contrast, a simpler model that did not utilize linguistic features only managed a 7% improvement.
  • Multi-scale Granularity: By using features based on "natural classes" of sounds, the model successfully learned rules at varying levels of specificity. It could recognize that a rule applies to all vowels generally, or specifically to the phoneme /u/ only when followed by a consonant, effectively solving the data sparsity problem.
  • Discovery of "Sound Laws": The model successfully identified well-known historical shifts. For example, it correctly learned the deletion of final /m/ and /s/ in the transition from Classical Latin to Vulgar Latin.
  • Interpretability: Unlike "black box" models, the log-linear weights allow researchers to see exactly which linguistic features (e.g., word-final position, alveolar consonants) are driving the changes, making the results directly useful for linguistic research.
Learned sound changes along the phylogenetic tree for the word /werbum/.

Because the model is fully generative, it doesn't just calculate similarity; it can actually predict what a word would look like in a different language or at a different point in time. This use of log-linear models successfully connects diachronic linguistics (how language changes over time) with synchronic linguistics (how language works at a single point in time).

Limitations and Future Work

While the framework is powerful, the authors note several limitations that present opportunities for future research:

  • Fixed Topology: The current model assumes the tree structure of the languages is already known. A natural next step is to use this phonological model to jointly infer the tree structure itself.
  • Cognate Identification: The model requires a pre-existing, curated list of cognates. Automating the discovery of these cognates from raw text remains a significant challenge.
  • Semantic Shift: The model focuses entirely on sound changes and does not account for how the meanings of words evolve over time.
  • Computational Complexity: The Monte Carlo EM approach is computationally intensive, which may limit its scalability to massive datasets without further optimization.

Summary updated

Abstract

We present a probabilistic approach to language change in which word forms are represented by phoneme sequences that undergo stochastic edits along the branches of a phylogenetic tree. Our framework combines the advantages of the classical comparative method with the robustness of corpus-based probabilistic models. We use this framework to explore the consequences of two different schemes for defining probabilistic models of phonological change, evaluating these schemes using the reconstruction of ancient word forms in Romance languages. The result is an efficient inference procedure for automatically inferring ancient word forms from modern languages, which can be generalized to support inferences about linguistic phylogenies.
BibTeX
@inproceedings{bouchardcote2007probabilistic,
  author = {Alexandre Bouchard-Côté and Percy Liang and Thomas L. Griffiths and Dan Klein},
  title = {{A Probabilistic Approach to Language Change}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2007},
  url = {https://openreview.net/forum?id=S1WN8Ob_bH},
}
Figure