Neural Network-Based Modeling of Phonetic Durations

Xizi Wei, Melvyn Hunt, Adrian SkillingInterspeech 2019

TL;DR

This paper introduces a novel deep neural network (DNN) approach for modeling phonetic durations, moving beyond traditional parametric methods. By treating duration prediction as a non-parametric distribution problem, the model effectively captures the influence of various linguistic features, offering a powerful tool for improving both speech synthesis (TTS) and automatic speech recognition (ASR) systems.

What Problem Does This Paper Address?

Accurately predicting the duration of individual speech sounds (phonemes) is crucial for natural-sounding speech synthesis and robust speech recognition. Previous work has often focused on single linguistic factors in isolation or used parametric models that struggle to capture the complex, non-linear relationships influencing duration. This paper aims to address this by developing a more sophisticated, data-driven model that can learn these relationships from large datasets and provide insights into which factors are most influential. The authors highlight that anomalous durations in training data can signal problems in TTS transcriptions or ASR alignments, making accurate duration modeling essential for data quality control.

Methods

The core of this work is a feedforward Deep Neural Network (DNN) that predicts a non-parametric distribution of durations for a given phoneme. Instead of outputting a single duration value, the DNN outputs probabilities for different duration bins. The model incorporates three types of input features:

  • Phoneme Identity: A one-hot encoding of the current phoneme.
  • Phonetic Context: Properties of adjacent phonemes (left and right).
  • Duration-Related Features:
    • Lexical Stress: A binary feature indicating if the current phoneme is in a stressed syllable, with additional features for adjacent phonemes.
  • Pre-pausal Lengthening: A feature indicating the distance to the next pause, with values representing 1 to 5 phonemes away (or >5).
  • Position in Syllable: A binary feature indicating if the current phoneme is a consonant preceding a vowel.
  • Word Predictability: Log probability scores from an n-gram language model, used as an inverse measure of word information content.
  • Speaking Rate: The ratio of the utterance's actual duration to its expected duration based on average phone durations.

The DNN uses ReLU activation for hidden layers and a softmax output layer. Training is optimized using the ADAM optimizer.

Datasets

The paper utilizes several in-house datasets:

  • SPK1: A dataset from a female American English speaker, recorded for TTS purposes, containing 64,795 utterances (33 hours).
  • SPK2: A dataset from a male American English speaker, also for TTS, with 27,550 utterances (13 hours).
  • SPK-ASR: A larger, less carefully controlled dataset intended for ASR, comprising 540,389 utterances from 535,556 speakers of all ages, including children.

Forced alignment was used to obtain reference durations for each phone in these datasets. SPK1 and SPK2 were used for speaker-dependent modeling, while SPK-ASR was used for speaker-independent modeling.

Key Insights and Ideas

  • Non-parametric duration modeling is powerful: Treating duration prediction as a distribution problem, rather than predicting a single value, allows the model to capture more nuanced temporal variations.
  • Linguistic features matter significantly: The DNN effectively learns the impact of features like lexical stress and pre-pausal lengthening, demonstrating their crucial role in shaping phonetic durations.
  • Context is king: Including phonetic context (neighboring phonemes) dramatically improves duration prediction accuracy compared to using only the current phoneme's identity.
  • Data quantity is critical: Speaker-dependent models trained on larger datasets (like SPK1) perform significantly better, highlighting the need for substantial amounts of speech data for optimal modeling.
  • Outliers reveal data issues: The model's ability to identify phonemes with low probability durations (outliers) is a valuable tool for detecting misalignments, disfluencies, or deviations from expected pronunciations in TTS and ASR training data.
  • Children's speech is more variable: The ASR dataset analysis shows a disproportionate number of outliers originating from children's speech, suggesting greater natural variability in their phonetic durations.

Key Results

  • Contextual improvement: Including ±1 phonetic context improved precision from 19% to 28.68% for speaker-dependent models.
  • Feature impact: Lexical stress showed the biggest positive effect on precision, followed by speaking rate and pre-pausal lengthening. Word predictability and syllable position had a weaker influence.
  • Best speaker-dependent model: A DNN with 3 hidden layers, 256 hidden units, and ±3 phonetic context achieved a precision of 35.67% and precision_3 of 89.88% on SPK1.
  • Speaker-independent performance: The speaker-independent model on the SPK-ASR dataset achieved a precision of 10.50% and precision_3 of 40.30%, indicating the challenge of modeling diverse speakers.
  • Outlier detection effectiveness: In the SPK1 TTS dataset, 48 out of 50 detected outliers were correctly identified as having bad alignments. For the ASR dataset, 24% of the top 50 outliers were from children, and 8% had bad alignments.

Limitations and Future Work

  • Speaker Independence: The performance of the speaker-independent model is significantly lower than speaker-dependent models, suggesting that further work is needed to capture inter-speaker variability, especially for features like stress and LM scores which were not yet incorporated.
  • Language Specificity: The current work is focused on American English. The authors speculate that duration modeling will be particularly useful for languages with phonemic length (e.g., Japanese, Finnish, Arabic).
  • Feature Engineering: While the model incorporates several key features, there may be other linguistic or acoustic factors that could further improve duration prediction.
  • Data Quality: The analysis of ASR outliers suggests that expressive speech or dictation mode can lead to low duration scores without invalidating the speech for ASR training, but disfluencies do. Future work could explore how to better distinguish between these types of anomalies.

Summary updated

BibTeX
@inproceedings{wei2019neural,
  author = {Xizi Wei and Melvyn Hunt and Adrian Skilling},
  title = {{Neural Network-Based Modeling of Phonetic Durations}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure