Abstract
The goal of generative phonology is to specify a formal system that explains the set of attested phonological strings in a language. Traditionally, this involves positing discrete Underlying Forms (UFs) and rules that transform them into Surface Forms (SFs). This paper implements the phonological generative system as a neural model differentiable end-to-end. In this model, UFs are continuous vectors in $\mathbb{R}^d$ discovered automatically through backpropagation. We compare several generative modes, evaluating their ability to predict phonological strings across 28 languages. Our results corroborate the necessity for UFs and their independence from SFs, demonstrating that a neural model can learn abstract linguistic representations at scale.
The Challenge of Traditional Phonology
In linguistics, generative phonology seeks to explain why the same morpheme (a unit of meaning) sounds different depending on its context. For example, the English past tense suffix -ed is realized as [t] in "talked," [d] in "saved," and [ɪd] in "acted." Traditionally, linguists explain this by positing an Underlying Form (UF)—a single "mental" representation—and a set of rules or constraints that transform that UF into a Surface Form (SF).
However, this traditional approach faces two major hurdles: it requires manual labor to "guess" UFs and rules, and the combinatorial search for discrete string-based UFs does not scale well to large vocabularies. This research asks whether we can automate this process using deep learning while respecting core linguistic tenets.
Methodology: Differentiable UFs
The authors propose Differentiable Generative Phonology, where UFs are continuous vectors rather than character strings. This allows the model to use backpropagation to learn the optimal mental representations for every morpheme. Three architectural variants were compared:
- Position-Independent UF (The "Linguistic" Model): The word's UF is a fixed vector (an average of its morpheme embeddings) fed into an LSTM decoder. This assumes the mental representation remains stable during production.
- Position-Dependent UF (The "Attention" Model): Uses an attention mechanism to look at different morphemes while generating each character of the SF, similar to standard Seq2Seq models.
- Joint Model (The "No-UF" Model): Skips the intermediate latent vector, directly conditioning the SF on the morphemes.
The latent-variable models are trained using Neural Variational Inference, optimizing the Evidence Lower Bound (ELBO) to maintain differentiability.
Emergent Linguistic Structures
A key finding of this approach is that the model discovers linguistic rules without explicit supervision. When visualizing learned Turkish UFs using t-SNE, the model naturally clustered stems based on vowel harmony. Back vowels and front vowels separated in vector space, proving the model captured the underlying phonological structure of the language.
Performance Across Languages
The models were evaluated on CELEX2 (phonetic forms for Dutch, English, and German) and UniMorph 2.0 (orthographic forms for 28 typologically diverse languages).
Surprisingly, the Position-Independent UF significantly outperformed the more complex attention-based model. In most NLP tasks, attention is superior; however, in phonology, having a single, stable underlying representation—as proposed by Chomsky and Halle in 1968—is computationally more effective for generalization. The "Joint Model" performed the worst, providing empirical evidence that an abstract intermediate representation is necessary to explain language patterns.
On the UniMorph dataset, the Position-Independent UF achieved an average accuracy of 93.0% across 28 languages. In CELEX2 experiments, it reduced the Edit Distance by 77% compared to the next best variant.
Conclusion
This differentiable approach successfully scales generative phonology to thousands of forms across diverse languages. While the continuous vectors currently lack the direct interpretability of IPA symbols and do not yet account for suprasegmentals like stress or tone, they provide a powerful, data-driven validation of classical linguistic theory. Future work aims to bridge the gap between these high-dimensional embeddings and discrete phonetic features.
Summary updated