Probabilistic Typology: Deep Generative Models of Vowel Inventories

Ryan Cotterell, Jason Eisner

Abstract

Linguistic typology studies the range of structures present in human language. The main goal of the field is to discover which sets of possible phenomena are universal, and which are merely frequent. For example, all languages have vowels, while most—but not all—languages have an /u/ sound. In this paper we present the first probabilistic treatment of a basic question in phonological typology: What makes a natural vowel inventory? We introduce a series of deep stochastic point processes, and contrast them with previous computational, simulation-based approaches. We provide a comprehensive suite of experiments on over 200 distinct languages.

The Problem: Defining "Natural" Inventories

Linguistic typology seeks to understand why certain language structures are universal or frequent. In phonology, a central observation is that while hundreds of vowel sounds are possible, most languages utilize a small inventory of 5 to 7 vowels, typically including [i], [a], and [u].

Traditional theories, such as Dispersion-Focalization Theory (DFT), posit two competing pressures:

  1. Focalization: Certain vowels are intrinsically "better" or more stable for production and perception.
  2. Dispersion: Vowels must be acoustically distinct from one another to ensure clear communication.

Previous computational models were largely non-probabilistic simulations. While they could generate plausible inventories, they lacked the ability to quantify the likelihood of real-world inventories or compare systems of different sizes. This research treats typology as a density estimation problem, seeking to model the "universal prior" of vowel inventories.

Methodology: Deep Stochastic Point Processes

The authors model vowel inventories using stochastic point processes, which define distributions over subsets of a base set of 53 unique IPA symbols. Three primary models were tested:

  • Bernoulli Point Process (BPP): A baseline model where vowels are chosen independently, capturing focalization but ignoring interaction.
  • Markov Point Process (MPP): A model that includes pairwise interaction terms to penalize vowels that are too close in acoustic space, explicitly modeling dispersion.
  • Determinantal Point Process (DPP): An elegant mathematical framework for diversity that uses a similarity matrix to favor "spread out" sets.

The "Deep" Twist

To improve these models, the authors utilize a deep neural network to learn a diffeomorphism—a smooth, invertible mapping—from raw acoustic formants ($F_1$ and $F_2$) to a latent metric space. This allows the model to discover a perceptual mapping where linguistic "repulsion" is more accurately represented than in raw frequency space.

The model learns a non-linear mapping (diffeomorphism) of the acoustic F1/F2 space to represent linguistic distances.

Key Results and Insights

The study utilized the Becker-Kristal (2010) corpus of 223 languages to evaluate performance via cross-entropy and a novel vowel system cloze task (predicting missing vowels in an inventory).

  • Dispersion is Quantifiable: Both MPP and DPP models significantly outperformed the BPP baseline, proving that dispersion is a measurable statistical force in typology.
  • Superiority of DPPs: The Determinantal Point Process generally performed best and offered better computational tractability than the MPP.
  • Non-linearity Matters: Models using deep neural transformations consistently outperformed linear ones, suggesting that the human ear's non-linear perception (like the Bark scale) is reflected in how languages organize their vowel systems.
Quantitative results showing that MPP and DPP models outperform the BPP baseline across cross-entropy and cloze tasks.

Discussion and Future Directions

By treating typology as density estimation, this framework allows linguists to quantify "naturalness" and make probabilistic predictions about unseen languages. However, the authors note several areas for future growth. The current model uses idealized average formant values and ignores features like vowel length, rounding, and nasality. Furthermore, future iterations could account for the evolutionary history of languages, rather than treating them as independent samples.

Ultimately, this work demonstrates that deep generative models can provide a principled, probabilistic foundation for long-standing questions in linguistic theory.

Summary updated

Abstract

Linguistic typology studies the range of structures present in human language. The main goal of the field is to discover which sets of possible phenomena are universal, and which are merely frequent. For example, all languages have vowels, while most---but not all---languages have an /u/ sound. In this paper we present the first probabilistic treatment of a basic question in phonological typology: What makes a natural vowel inventory? We introduce a series of deep stochastic point processes, and contrast them with previous computational, simulation-based approaches. We provide a comprehensive suite of experiments on over 200 distinct languages.
BibTeX
@inproceedings{cotterell2017probabilistic,
  author = {Ryan Cotterell and Jason Eisner},
  title = {{Probabilistic Typology: Deep Generative Models of Vowel Inventories}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2017},
  url = {https://openreview.net/forum?id=ryZaRneObr},
}
Figure