Lombard Speech Synthesis Using Transfer Learning in a Tacotron Text-to-Speech System

Bajibabu Bollepalli, Lauri Juvela, Paavo AlkuInterspeech 2019

TL;DR

This paper tackles the challenge of synthesizing Lombard speech, a style humans adopt for intelligibility in noisy environments, which is difficult to collect large datasets for. They propose a transfer learning approach to adapt a standard text-to-speech (TTS) system to Lombard style, achieving better results than previous methods, especially when combined with the WaveNet vocoder.

What Problem Does This Paper Address?

Modern text-to-speech (TTS) systems, particularly those based on sequence-to-sequence (Seq2Seq) models like Tacotron, excel at generating natural-sounding speech. However, these systems are data-hungry, requiring extensive datasets for training. This becomes a significant hurdle when dealing with specialized speaking styles, such as Lombard speech. Lombard speech is characterized by increased vocal effort and is naturally adopted by humans to improve intelligibility in noisy environments. Recording large corpora of Lombard speech is challenging due to the effort involved for speakers. Consequently, adapting TTS systems to this style using data-driven methods is difficult. The paper aims to bridge this gap by leveraging existing large datasets of normal speech to effectively adapt a TTS system to the Lombard speaking style, even with limited Lombard speech data.

Methods

The core of the paper's methodology lies in transfer learning (TL) applied to a Tacotron-based Seq2Seq TTS system. The authors employ a fine-tuning approach in two stages:

  1. Initial Training: A Seq2Seq TTS model is first trained on a large dataset of normal speech from a female speaker (named Nancy). This establishes a robust foundation for general speech synthesis.
  2. Fine-tuning for Speaker Adaptation: The pre-trained model is then fine-tuned on normal speech from a male speaker (named Nick) with a smaller dataset. This step adapts the model to the target speaker's voice characteristics.
  3. Fine-tuning for Style Adaptation: Finally, the speaker-adapted model is further fine-tuned using a limited dataset of Lombard speech from the same male speaker (Nick). This crucial step adapts the model to the Lombard speaking style.

The system predicts WORLD vocoder parameters (mel-generalized cepstrum, fundamental frequency, and band aperiodicity) as acoustic features. They also experiment with predicting mel-spectrograms as acoustic features. For waveform generation, they compare the traditional WORLD vocoder with the neural WaveNet vocoder.

Datasets

  • Blizzard Challenge 2011 speech corpus: Used for training the initial Seq2Seq TTS model on normal speech from a female speaker (Nancy). This dataset comprises approximately 12,000 utterances (around 16 hours of speech).
  • Hurricane Challenge speech data: Used for adaptation. This dataset includes:
  • Normal speech from a male speaker (Nick): 2,592 utterances (around 2 hours of speech).
  • Lombard speech from the same male speaker (Nick): 720 utterances (around 30 minutes of speech).

All data was sampled at 16kHz. The data was partitioned into training, validation, and testing sets.

Key Insights and Ideas

  • Transfer learning is crucial for low-resource speaking styles: The paper strongly demonstrates that TL is an effective strategy for adapting TTS systems to challenging speaking styles like Lombard speech, where data scarcity is a major issue.
  • Seq2Seq models benefit from pre-training: Training a large Seq2Seq model on a general dataset (Nancy's normal speech) provides a strong starting point, enabling better generalization and alignment when fine-tuned on smaller, style-specific datasets.
  • WaveNet vocoder enhances Lombard speech quality: The WaveNet vocoder, when conditioned on mel-spectrograms, significantly improves the intelligibility and naturalness of synthesized Lombard speech compared to the traditional WORLD vocoder.
  • Mel-spectrograms as features with WaveNet are superior: System S5, which uses mel-spectrograms as features and the WaveNet vocoder, consistently outperformed other systems in both subjective and objective evaluations for Lombard speech synthesis.

Key Results

  • Subjective Evaluation (Style Similarity): System S5 (Seq2Seq-TTS with mel-spectrograms and WaveNet) was rated highest for sounding most Lombard-like, outperforming baseline systems and those using the WORLD vocoder. Systems using WaveNet generally performed better than those using WORLD.
  • Subjective Evaluation (Comparison Category Rating - CCR): System S5 achieved significantly better naturalness scores than the baseline (S1) and the LSTM-based system (S3).
  • Instrumental Intelligibility Evaluation (SIIBGauss): System S5 achieved the highest intelligibility scores across various Signal-to-Noise Ratios (SNRs), indicating superior performance compared to the baseline (S1) and other systems. The WaveNet vocoder consistently led to higher intelligibility scores.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the Lombard speech data for adaptation was limited (30 minutes).
  • While successful, the fine-tuning process still requires careful parameter tuning.
  • Future Work:
    • The authors suggest conducting more extensive subjective evaluations.
  • They propose training the WaveNet and Seq2Seq-TTS models in a single, end-to-end pipeline for potentially further improvements.

Summary updated

BibTeX
@inproceedings{bollepalli2019lombard,
  author = {Bajibabu Bollepalli and Lauri Juvela and Paavo Alku},
  title = {{Lombard Speech Synthesis Using Transfer Learning in a Tacotron Text-to-Speech System}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure