Acoustic barycenters as exemplar production targets

Frederic Mailhot, Cassandra L. JacobsNAACL 2024

TL;DR

This paper introduces MNEMORPHON, a novel exemplar-based model for speech production that leverages dynamic time warping (DTW) and DTW barycenter averaging (DBA) to handle variable-duration speech tokens. The model successfully generates acoustic representations of words that retain essential phonetic characteristics, demonstrating a promising step towards more natural and flexible speech synthesis.

What Problem Does This Paper Address?

Producing speech from memory, as humans do, is incredibly complex. One major challenge is dealing with the inherent variability in how we say things – the same word can be spoken faster, slower, or with slight variations in pronunciation each time. Traditional exemplar-based models, which rely on direct comparison to stored examples, struggle to handle this temporal variability and to generalize from learned experiences to produce novel utterances. Existing approaches often require ad-hoc modifications or are computationally intractable. This paper aims to bridge this gap by developing a principled method for exemplar-based speech production that can robustly handle variable-duration speech tokens and generalize effectively.

Methods

The core of MNEMORPHON's approach lies in its use of advanced time-series analysis techniques:

  • Dynamic Time Warping (DTW): This algorithm is used to find the optimal alignment between speech segments (tokens) of potentially different lengths. By "warping" the time axis, DTW allows for a meaningful comparison of acoustic features even when utterances are not perfectly synchronized. This is a standard technique in speech recognition but is applied here in a novel way for production.
  • DTW Barycenter Averaging (DBA): To generate an "average" speech token from a collection (an "exemplar cloud"), the authors employ DBA. This algorithm iteratively computes a barycenter (an average sequence) that minimizes the sum of DTW distances to all sequences in the cloud. This is a more principled approach to averaging variable-length sequences compared to simpler methods.
  • Mel-scaled Spectrograms: Speech tokens are represented as mel-scaled spectrograms, which capture the spectral characteristics of the audio over time. The parameters for these spectrograms were chosen to be compatible with a pre-trained neural vocoder for subsequent audio synthesis and evaluation.

Datasets

The experiments were conducted using a corpus of Turkish speech from 120 speakers. This corpus included audio recordings, standard Turkish orthography, and phonemic transcriptions. Metadata such as speaker ID, gender, and recording details were also available. For the vowel classification experiment, vowels were extracted from this corpus, resulting in a dataset of 82,360 samples.

Key Insights and Ideas

  • DTW and DBA as a Foundation for Production: The paper convincingly demonstrates that DTW and DBA provide a robust mathematical framework for handling temporal variability in exemplar-based speech production, overcoming a significant hurdle in the field.
  • Averaging Can Be Noisy, but Meaningful: While increasing the number of exemplars in the "cloud" for averaging can lead to "blurrier" outputs (loss of fine temporal detail), the core acoustic and phonetic characteristics are still preserved. This suggests that even averaged representations can be phonetically informative.
  • Cloud Composition Matters: The dispersion of tokens within the exemplar cloud significantly impacts the quality of the generated output. Tightly clustered clouds (e.g., using only the nearest neighbors in DTW distance) produce cleaner results than widely dispersed clouds.
  • Indexical Information Can Improve Outputs: Constraining the exemplar cloud selection using contextual information, such as speaker gender, can lead to cleaner and more representative outputs, hinting at the potential for incorporating richer sociophonetic information.
  • Recoverable Categorical Information: Despite the noise, the generated spectrograms contain enough phonetic information for a neural network to classify vowels as front or back with high accuracy, suggesting that the model captures essential categorical distinctions.

Key Results

  • The MNEMORPHON model, using DTW and DBA, successfully generates mel-scaled spectrograms of words that retain essential acoustic/phonetic characteristics.
  • Qualitative evaluation showed that increasing cloud size leads to blurrier spectrograms, but the fundamental structure remains.
  • Constraining the exemplar cloud by gender information improved the quality of the generated spectrograms.
  • A convolutional neural network classifier trained on these generated spectrograms achieved an accuracy of 0.873 in distinguishing front and back vowels, demonstrating that the model captures phonetically relevant information.

Limitations and Future Work

  • Output Noise: The authors acknowledge that the generated outputs can be "noisy," especially with larger exemplar clouds. This is a trade-off for generalization.
  • Binarized Gender: The use of binarized gender for cloud constraint is noted as a simplification, and future work should explore a wider range of gender identities or purely phonetic features.
  • Future Directions: The authors plan to explore further constraints on cloud construction (e.g., speaker identity, dialect, speech rate). They also aim to extend the model to account for productive morphophonological alternations (like Turkish vowel harmony) and to conduct experiments with more accessible datasets for reproducibility. The ultimate goal is to contribute to a fully articulated theory of exemplar-based knowledge.

Summary updated

BibTeX
@inproceedings{mailhot2024acoustic,
  author = {Frederic Mailhot and Cassandra L. Jacobs},
  title = {{Acoustic barycenters as exemplar production targets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure