Introduction
In the field of linguistics, a fundamental distinction exists between phonetics—the physical, acoustic properties of speech—and phonology—the abstract mental grammar that governs how sounds pattern together. For instance, in English, vowels are predictably nasalized before nasal consonants (e.g., the 'a' in ban), a rule that does not change word meaning. Conversely, in French, nasality is contrastive: the difference between beau (oral) and bon (nasal) distinguishes two different words.
This study, presented at ACL 2023, investigates whether Generative Adversarial Networks (GANs) learn these abstract phonological rules or merely mimic raw phonetic sounds. Specifically, the researchers explored whether GANs learn features like nasality as distinct, controllable latent variables.
Methodology
The researchers utilized the Categorical InfoWaveGAN (ciwGAN) architecture. This model is designed to discover categorical variables in an unsupervised manner by maximizing the mutual information between a subset of latent variables and the generated output.
To analyze the internal representations, the team employed three primary techniques:
- Latent Variable Manipulation: Systematically altering values in the latent space to observe acoustic changes in generated audio.
- Supervised Probing (nasalDNN): Training a separate 1D Convolutional Neural Network to detect nasality in GAN-generated audio, providing an objective metric for latent control.
- Regression Analysis: Using chi-square scores to identify which of the 100 latent variables correlated most strongly with nasality.
Datasets
The study utilized two primary speech corpora to compare cross-linguistic behavior:
- TIMIT (English): Used to extract sequences of vowels followed by stops (VT) or nasals (VN).
- SIWIS (French): Used to extract oral and nasal vowel sequences (VT, VN, $\tilde{V}T$, $\tilde{V}N$).
- Balanced Datasets: Artificial versions of these datasets were created to test if the frequency of specific sounds (like nasal vowels) influenced the model's categorical learning.
Key Results and Discussion
The "One-to-One" Mapping Myth
Previous research suggested that ciwGAN could learn linguistic features in a transparent, one-to-one way (one "knob" for one feature). However, this study found significant interactive effects. Multiple variables often work in tandem; changing a "nasal" variable might only produce the desired effect if another latent variable is within a specific range.
Language-Specific Encoding
The GAN's learning was found to be highly sensitive to the phonological structure of the training language. In English-trained models, latent variables primarily controlled the presence of the nasal consonant [n], with vowel nasality appearing as a secondary, dependent effect. In contrast, French-trained models successfully identified independent variables for nasal vowels, reflecting the contrastive nature of French phonology.
As shown in the heatmaps for French, variables such as $z88$ and $z91$ demonstrate a clear ability to toggle vowel nasality. This confirms that the model can learn to represent phonological contrasts when they are meaningful in the input data.
Frequency-Driven Phonology
By training on balanced datasets, the authors demonstrated that "phonologization" in neural models is largely a byproduct of statistical distribution. When the English model was trained on a dataset with an artificially high variety of nasal sounds, it began to generate independent nasal vowels—a phenomenon not found in natural English. This suggests that the model's internal categories are driven by the density of examples in the training data.
Conclusion
While GANs can learn to represent abstract linguistic features, their internal representations are more complex and interactive than simple one-to-one mappings. The study highlights that the "phonological" behavior of a model is a reflection of both the language's structural constraints and the statistical frequency of sounds in the training set. Future work will focus on using advanced methods like eigendecomposition to further disentangle these multi-variable relationships and developing broader phonological benchmarks.
Summary updated