Parallel vs. Non-Parallel Voice Conversion for Esophageal Speech

Luis Serrano, Sneha Raman, David Tavarez, Eva Navas, Inma HernaezInterspeech 2019

TL;DR

This paper explores two voice conversion (VC) approaches to improve the intelligibility and naturalness of esophageal speech (ES), a voice produced by individuals without a larynx. While a parallel LSTM-based system improved intelligibility, a non-parallel Phonetic Posteriorgram (PPG)-based system, designed to work without paired data, surprisingly degraded intelligibility, suggesting that current PPG-based methods may not be robust enough for highly distorted speech like ES.

What Problem Does This Paper Address?

Individuals who have undergone a laryngectomy (surgical removal of the larynx) often resort to producing alaryngeal speech, such as esophageal speech (ES). ES is produced by swallowing air and expelling it through the pharyngo-esophageal segment, which acts as a substitute for vocal folds. This production method, however, significantly diminishes speech naturalness and intelligibility, hindering communication. Furthermore, the acoustic characteristics of ES pose a challenge for automatic speech recognition (ASR) systems, which are increasingly integrated into human-computer interaction. This paper aims to address this by applying voice conversion techniques to transform ES into a more natural and intelligible alaryngeal voice, specifically targeting a healthy speaker's voice as the target.

Methods

The paper compares two voice conversion systems:

  • LSTM-based spectral conversion (Parallel): This system uses a Long Short-Term Memory (LSTM) neural network trained on an aligned dataset of parallel source (esophageal) and target (healthy) speech. It learns to map Mel-cepstral coefficients (MCEP) and their derivatives from the source to the target. This is a more traditional approach that requires paired data.
  • PPG-based spectral conversion (Non-Parallel): This novel approach leverages Phonetic Posteriorgrams (PPGs) extracted from an esophageal-specific ASR system. The core idea is that PPGs can represent phonetic information independently of the speaker's voice, allowing for conversion without parallel data. A Deep Bidirectional LSTM (DBLSTM) network is trained to predict target acoustic parameters from these PPGs. A key challenge addressed here is adapting the ASR system to the unique characteristics of esophageal speech, which differs significantly from typical laryngeal speech.

Both systems also incorporate a fundamental frequency (fo) estimation network to generate the pitch contour for the converted speech.

Datasets

  • Healthy Speaker Dataset: A dataset of 3995 recordings from a single male speaker was used as the target for voice conversion.
  • Esophageal Speaker Dataset: This dataset comprised recordings of 100 phonetically balanced sentences from 30 esophageal speakers (25 male, 4 female, and one tracheoesophageal male). One male esophageal speaker was selected as the source.
  • ASR Training Data: For the esophageal ASR system, recordings from 29 esophageal speakers (2610 sentences) were used.

Key Insights and Ideas

  • The challenge of non-parallel VC for highly distorted speech: The paper highlights that while PPG-based VC is promising for its ability to work without parallel data, its effectiveness can be severely hampered when the source speech is highly distorted, as is the case with esophageal speech. The acoustic characteristics of ES are so different from healthy speech that the adapted ASR system still produces PPGs that, when used for conversion, lead to a degradation in intelligibility.
  • Importance of domain-specific ASR for PPG-based VC: The authors emphasize the necessity of developing a specialized ASR system for esophageal speech. A standard, speaker-independent ASR system trained on healthy speech is insufficient because it cannot accurately capture the phonetic information from ES, leading to poor PPGs and subsequent conversion.
  • Intelligibility vs. Naturalness Trade-off: The objective evaluation (WER) and subjective evaluation (preference test) revealed a discrepancy. The LSTM-based parallel system improved intelligibility (lower WER), but listeners did not significantly prefer it, suggesting that the converted speech might sound less natural. Conversely, the PPG-based system did not improve intelligibility and was not strongly preferred over the original ES.
  • Small data limitations: The authors acknowledge that the limited amount of data available for training the different systems might be a significant factor contributing to the suboptimal performance of both conversion methods.

Key Results

  • Objective Evaluation (WER):
    • Original Esophageal Speech: 56.93% WER
    • Healthy Target Speech: 11.88% WER
    • LSTM Converted Speech: 40.58% WER (Improvement of ~16 percentage points over original ES)
    • PPG Converted Speech: 57.91% WER (Slightly worse than original ES)
  • Objective Evaluation (MCD): Both the LSTM and PPG conversion methods reduced the spectral distance to the target speech similarly, with no significant difference between them.
  • Subjective Evaluation (Preference Test):
    • Original ES was preferred over LSTM converted speech by a statistically significant margin.
  • No significant preference was observed between original ES and PPG converted speech.
  • Listeners showed a slight, non-significant preference for the PPG system when comparing the two converted versions.
  • Detailed preference breakdown showed that for PPG vs. original, 48.9% preferred original, and 37.5% preferred PPG. For LSTM vs. original, 50.7% preferred original, and 34.3% preferred LSTM.

Limitations and Future Work

  • Limitations:
    • The PPG-based system's performance was significantly degraded due to the highly distorted nature of esophageal speech, indicating that current PPG extraction and conversion methods may not be robust enough for such challenging cases.
  • The limited amount of training data available for both systems is identified as a major factor contributing to the poor overall performance.
  • Future Work: The paper implicitly suggests that further research is needed to improve the robustness of PPG-based VC for highly distorted speech. This could involve developing more advanced ASR systems tailored to alaryngeal speech or exploring alternative non-parallel VC techniques. The authors also point to the need for larger datasets to train more effective models.

Summary updated

BibTeX
@inproceedings{serrano2019parallel,
  author = {Luis Serrano and Sneha Raman and David Tavarez and Eva Navas and Inma Hernaez},
  title = {{Parallel vs. Non-Parallel Voice Conversion for Esophageal Speech}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure