Encoder-decoder models for latent phonological representations of words

Cassandra L. Jacobs, Frédéric MailhotACL 2019

TL;DR

This paper introduces a novel approach using encoder-decoder neural networks to learn latent phonological representations of words. These learned representations prove effective in predicting phonetic durations and correlate with established linguistic measures, suggesting their utility for modeling psycholinguistic phenomena.

What Problem Does This Paper Address?

Representing the phonological structure of words is a fundamental challenge in linguistics and natural language processing. Traditional methods often rely on discrete phonetic segments or simplified representations that struggle to capture the nuances of spoken language, such as variability in pronunciation and the sequential nature of speech. This paper aims to bridge this gap by exploring whether modern sequence-to-sequence models can learn rich, latent phonological representations that are both linguistically valid and computationally useful for downstream tasks.

Methods

The core of this work is the application of recurrent encoder-decoder (specifically, LSTM-based) neural networks. These models are trained on phonetic transcriptions of words. Two main tasks are explored:

  • Normative Decoder: The model learns to predict the standard, dictionary pronunciation of a word from its observed phonetic realization.
  • Observed Decoder (Seq2Seq Autoencoder): The model learns to reproduce the input phonetic sequence exactly.

The encoder component of these trained models generates a fixed-dimensional latent representation for each word. This latent representation is then used for downstream tasks, such as predicting phonetic duration. The authors also compare their approach to several baseline methods, including word embeddings, bag-of-phones, bag-of-n-phones, PCA on bag-of-n-phones, doc2vec, and the normative decoder's output.

Datasets

The primary dataset used is the Buckeye corpus, a collection of monologues containing approximately 300,000 words with phonetic transcriptions. The authors used a randomly selected 80/20 train/test split of this corpus. For word embeddings, they utilized embeddings trained on the Fisher corpus.

Key Insights and Ideas

  • Seq2Seq Models Capture Phonological Structure: Encoder-decoder models, particularly the "Observed Decoder," can learn latent representations that effectively capture aspects of a word's phonological structure.
  • Latent Representations Predict Phonetic Duration: The learned representations are shown to be predictive of whole-word phonetic duration, outperforming several traditional linguistic measures.
  • Encoding of Serial Order: The unidirectional nature of the LSTM encoder leads to representations that strongly encode the final segment of a word, a pattern visualized in t-SNE plots.
  • Flexibility in Input: The models can learn from both observed, variable phonetic realizations and idealized, normative pronunciations, offering flexibility for different data availability scenarios.
  • Handling OOV Words: The fixed dimensionality of the latent representations allows for generating representations for out-of-vocabulary (OOV) and non-words, which is a significant advantage over methods that rely on direct lexical lookup.

Key Results

  • The Observed Decoder model's latent representations achieved the highest R² value (0.181) in predicting log phonetic duration when combined with bag-of-phones and word embeddings, significantly outperforming baseline models.
  • The Observed Decoder alone also performed well, achieving an R² of 0.149, which was comparable to or better than most baseline methods.
  • The learned representations showed strong correlations with established linguistic measures like phonotactic probability (positive correlation) and phonological neighborhood density (negative correlation with mean similarity, non-linear with standard deviation).
  • t-SNE visualizations of the word vectors revealed that clusters largely prioritize word-final information, especially the last segment, due to the unidirectional nature of the recurrent layers.

Limitations and Future Work

  • Normative Decoder Performance: The "Normative Decoder" model showed slightly worse performance than the "Observed Decoder." The authors suggest this might be due to the more difficult learning problem for the normative task or differences in precision. Future work could explore this further.
  • Bi-directional LSTMs: The authors hypothesize that using a bi-directional LSTM encoder could lead to further advances in creating psycholinguistically predictive word representations.
  • Contextual Predictability: While phonetic duration is known to be sensitive to contextual predictability, the paper primarily focuses on phonotactic probability and phonological overlap. Further investigation into how contextual predictability is encoded in the latent representations is suggested.
  • Meta-model for Similarity: The authors note that a meta-model will be necessary to determine what factors drive a word's mean lexicon-similarity value, as observed in Figure 4.
  • Word Length Correlation: Reviewers pointed out that word length in segments is correlated with the observed properties (e.g., frequency, similarity clusters). The authors acknowledge this as a useful baseline that they plan to explore further.

Summary updated

BibTeX
@inproceedings{jacobs2019encoder,
  author = {Cassandra L. Jacobs and Frédéric Mailhot},
  title = {{Encoder-decoder models for latent phonological representations of words}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure