Differentiable Generative Phonology

Shijie Wu, Edoardo Maria Ponti, Ryan CotterellarXiv 2021 · OpenReview

Abstract

The goal of generative phonology is to specify a formal system that explains the set of attested phonological strings in a language. Traditionally, this involves positing discrete Underlying Forms (UFs) and rules that transform them into Surface Forms (SFs). This paper implements the phonological generative system as a neural model differentiable end-to-end. In this model, UFs are continuous vectors in $\mathbb{R}^d$ discovered automatically through backpropagation. We compare several generative modes, evaluating their ability to predict phonological strings across 28 languages. Our results corroborate the necessity for UFs and their independence from SFs, demonstrating that a neural model can learn abstract linguistic representations at scale.

The Challenge of Traditional Phonology

In linguistics, generative phonology seeks to explain why the same morpheme (a unit of meaning) sounds different depending on its context. For example, the English past tense suffix -ed is realized as [t] in "talked," [d] in "saved," and [ɪd] in "acted." Traditionally, linguists explain this by positing an Underlying Form (UF)—a single "mental" representation—and a set of rules or constraints that transform that UF into a Surface Form (SF).

However, this traditional approach faces two major hurdles: it requires manual labor to "guess" UFs and rules, and the combinatorial search for discrete string-based UFs does not scale well to large vocabularies. This research asks whether we can automate this process using deep learning while respecting core linguistic tenets.

Methodology: Differentiable UFs

The authors propose Differentiable Generative Phonology, where UFs are continuous vectors rather than character strings. This allows the model to use backpropagation to learn the optimal mental representations for every morpheme. Three architectural variants were compared:

  1. Position-Independent UF (The "Linguistic" Model): The word's UF is a fixed vector (an average of its morpheme embeddings) fed into an LSTM decoder. This assumes the mental representation remains stable during production.
  2. Position-Dependent UF (The "Attention" Model): Uses an attention mechanism to look at different morphemes while generating each character of the SF, similar to standard Seq2Seq models.
  3. Joint Model (The "No-UF" Model): Skips the intermediate latent vector, directly conditioning the SF on the morphemes.

The latent-variable models are trained using Neural Variational Inference, optimizing the Evidence Lower Bound (ELBO) to maintain differentiability.

Emergent Linguistic Structures

A key finding of this approach is that the model discovers linguistic rules without explicit supervision. When visualizing learned Turkish UFs using t-SNE, the model naturally clustered stems based on vowel harmony. Back vowels and front vowels separated in vector space, proving the model captured the underlying phonological structure of the language.

t-SNE visualization of learned Turkish UFs, showing automatic clustering by vowel harmony.

Performance Across Languages

The models were evaluated on CELEX2 (phonetic forms for Dutch, English, and German) and UniMorph 2.0 (orthographic forms for 28 typologically diverse languages).

Surprisingly, the Position-Independent UF significantly outperformed the more complex attention-based model. In most NLP tasks, attention is superior; however, in phonology, having a single, stable underlying representation—as proposed by Chomsky and Halle in 1968—is computationally more effective for generalization. The "Joint Model" performed the worst, providing empirical evidence that an abstract intermediate representation is necessary to explain language patterns.

Performance metrics (Accuracy, Mean Levenshtein Distance, Negative Log-Likelihood) across 28 languages for the three model variants.

On the UniMorph dataset, the Position-Independent UF achieved an average accuracy of 93.0% across 28 languages. In CELEX2 experiments, it reduced the Edit Distance by 77% compared to the next best variant.

Conclusion

This differentiable approach successfully scales generative phonology to thousands of forms across diverse languages. While the continuous vectors currently lack the direct interpretability of IPA symbols and do not yet account for suprasegmentals like stress or tone, they provide a powerful, data-driven validation of classical linguistic theory. Future work aims to bridge the gap between these high-dimensional embeddings and discrete phonetic features.

Summary updated

Abstract

The goal of generative phonology, as formulated by Chomsky and Halle (1968), is to specify a formal system that explains the set of attested phonological strings in a language. Traditionally, a collection of rules (or constraints, in the case of optimality theory) and underlying forms (UF) are posited to work in tandem to generate phonological strings. However, the degree of abstraction of UFs with respect to their concrete realizations is contentious. As the main contribution of our work, we implement the phonological generative system as a neural model differentiable end-to-end, rather than as a set of rules or constraints. Contrary to traditional phonology, in our model, UFs are continuous vectors in $\mathbb{R}^d$, rather than discrete strings. As a consequence, UFs are discovered automatically rather than posited by linguists, and the model can scale to the size of a realistic vocabulary. Moreover, we compare several modes of the generative process, contemplating: i) the presence or absence of an underlying representation in between morphemes and surface forms (SFs); and ii) the conditional dependence or independence of UFs with respect to SFs. We evaluate the ability of each mode to predict attested phonological strings on 2 datasets covering 5 and 28 languages, respectively. The results corroborate two tenets of generative phonology, viz. the necessity for UFs and their independence from SFs. In general, our neural model of generative phonology learns both UFs and SFs automatically and on a large-scale.
BibTeX
@misc{wu2021differentiable,
  author = {Shijie Wu and Edoardo Maria Ponti and Ryan Cotterell},
  title = {{Differentiable Generative Phonology}},
  year = {2021},
  eprint = {2102.05717},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.05717},
}
Figure