Learning an artificial F0-contour for ALT speech

Anna Katharina Fuchs, Martin HagmüllerInterspeech 2012

TL;DR

This paper presents a novel machine learning approach to automatically generate a more natural-sounding fundamental frequency (F0) contour for artificial larynx (ALT) speech. By using Gaussian Mixture Models (GMMs) trained on both healthy and ALT speech, the system learns to mimic the prosodic variations of natural speech, significantly improving the intelligibility and naturalness of alaryngeal speech.

What Problem Does This Paper Address?

For individuals who have undergone a total laryngectomy, often due to cancer, the ability to speak is lost as the larynx is removed. While several methods exist to restore speech, such as esophageal voice, tracheo-esophageal voice, and the Artificial Larynx Transducer (ALT), the resulting speech often lacks naturalness and intelligibility. The ALT, a common device, produces speech with a constant fundamental frequency (F0), making it sound robotic and monotonous. Existing methods to introduce F0 variation are often manual or provide limited control. This paper addresses the need for an automatic, low-complexity method to generate a dynamic F0 contour for ALT speech, aiming to enhance its prosody and make it more human-like.

Methods

The core of the proposed method is the use of Gaussian Mixture Models (GMMs) to learn an artificial F0 contour. GMMs are probabilistic models capable of representing complex probability distributions. The authors train a GMM using Mel-frequency cepstral coefficients (MFCCs) extracted from ALT speech and the corresponding F0 values from healthy speech. This allows the model to learn the relationship between spectral features and F0.

During the test phase, MFCCs are extracted from an unknown ALT speech utterance. The trained GMM is then used to estimate the F0 contour for this utterance. The process involves identifying the nearest Gaussian component in the GMM to the input MFCC vector and using this to predict the F0. This approach is designed to be speaker- and gender-dependent, meaning separate models are trained for different individuals. The method is intended for real-time application, though the current implementation is offline.

Datasets

The authors created a custom speech database for their experiments. This database consists of speech material recorded from one female and one male speaker. For each speaker, 97 sentences were recorded twice: once with their natural, healthy voice (HE) and once using a Servox electronic larynx (ALT). The recorded sentences cover various types, including questions, declarations, and imperatives. The average sentence length was approximately 5 seconds.

Key Insights and Ideas

  • The vocal tract is key: Even without a larynx, the vocal tract remains largely intact. This paper leverages the idea that spectral features captured from speech produced with an ALT still contain valuable information about speech prosody, which can be used to infer natural F0 variations.
  • GMMs for prosody learning: Gaussian Mixture Models prove to be a suitable tool for learning the complex mapping between spectral features and fundamental frequency, enabling the generation of a dynamic F0 contour.
  • Automatic learning is crucial: The reliance on manual control in previous F0 modification techniques is a significant barrier to practical use. An automatic learning approach, as presented here, is a substantial step towards more accessible and effective speech enhancement for ALT users.
  • Prosody impacts perception: Even small variations in the F0 contour can significantly improve the perceived naturalness and intelligibility of alaryngeal speech, as suggested by informal listening tests and numerical evaluations.

Key Results

The paper reports significant improvements in the quality of ALT speech. For a specific sentence from the male speaker, the correlation coefficient between the natural F0 and the original ALT F0 was 0.41, while the correlation with the estimated F0 reached a much higher value of 0.91.

In terms of absolute mean error, measured in semitones (midi), the original ALT F0 had a mean error of 12.2 semitones compared to the natural F0. The proposed method reduced this error to a mere 0.7 semitones. These results were further validated through a 4-fold cross-validation, showing consistent improvements for both male and female speakers. The tables and figures in the paper clearly illustrate the substantial reduction in error achieved by the proposed method.

Limitations and Future Work

The authors acknowledge that informal perceptual evaluations were used to verify the numerical results, and that formal listening tests are time and cost-consuming. While the method shows promise, it is noted that the current implementation is offline, and real-time application is a future goal.

Future research directions include:

  • Improving the voiced/unvoiced decision for ALT speech.
  • Enhancing gender variation to make female voices more distinctly perceived as female.
  • Further improving overall sound quality while preserving voice identity.
  • Allowing users to have individual voice characteristics.

Summary updated

BibTeX
@inproceedings{fuchs2012learning,
  author = {Anna Katharina Fuchs and Martin Hagmüller},
  title = {{Learning an artificial F0-contour for ALT speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure