The use of air-pressure sensor in electrolaryngeal speech enhancement based on statistical voice conversion

Keigo Nakamura, Tomoki Toda, Hiroshi Saruwatari, Kiyohiro ShikanoInterspeech 2010

TL;DR

This paper introduces a novel speaking-aid system for laryngectomees that enhances electrolaryngeal (EL) speech by incorporating an air-pressure sensor to control fundamental frequency (Fo). By combining spectral and Fo information from this sensor-enhanced EL speech, the system significantly improves the naturalness and accuracy of converted speech compared to previous methods.

What Problem Does This Paper Address?

Laryngectomy, the surgical removal of the larynx, often necessitates the use of an electrolarynx (EL) to restore speech. However, EL speech is typically unnatural due to its fixed, built-in sound source. While previous attempts at voice conversion (VC) have improved EL speech, a key challenge remains in accurately estimating natural fundamental frequency (Fo) contours. Existing systems struggle to derive rich Fo information solely from the spectral characteristics of EL speech, leading to less-than-ideal naturalness. This paper aims to bridge this gap by leveraging an air-pressure sensor to provide more controllable and informative Fo data for the VC system.

Methods

The core of the proposed system is a statistical voice conversion (VC) technique based on Gaussian Mixture Models (GMMs). The paper builds upon their previous GMM-based VC system, which converts EL speech to normal speech. The novelty lies in the integration of an air-pressure sensor that allows laryngectomees to control the Fo of their EL speech by modulating their breathing air.

The GMM-based VC method involves two main parts:

  • Training Part: A GMM is trained to model the joint probability distribution of source (EL speech) and target (normal speech) feature vectors. This includes spectral features and, crucially for this work, Fo information.
  • Conversion Part: Given a new EL speech utterance, the system estimates the target feature sequence (including Fo) that maximizes the conditional probability given the source features.

The key innovation here is the input to the VC system. Instead of relying solely on spectral information from standard EL speech, the proposed system uses EL(air) speech, which is generated with the air-pressure sensor. This allows the system to utilize both the spectral information and the intentionally controlled Fo contours from the EL(air) speech for more accurate and natural conversion. The paper also describes a method for constructing segmental feature vectors that combine spectral and Fo information, as illustrated in Figure 3.

Datasets

The paper describes the experimental setup using:

  • Training Data: 50 phoneme-balanced sentences recorded by both a laryngectomized speaker (using EL, EL(small), and EL(air) speech) and a target normal speech speaker.
  • Test Data: 30 newspaper articles recorded by the same speakers.

The authors also mention that the laryngectomized speaker trained to control Fo using the air-pressure sensor for a total of seven weeks.

Key Insights and Ideas

  • Air-Pressure Sensor for Controllable Fo: The air-pressure sensor provides a direct and intuitive mechanism for laryngectomees to influence the Fo of their EL speech, moving beyond the limitations of fixed EL sound sources.
  • Synergy of Spectral and Fo Information: The most significant insight is that combining spectral features with the controlled Fo information from EL(air) speech leads to substantially better VC performance than using spectral information alone.
  • Improved Naturalness through Fo Control: While EL(air) speech itself might not have perfectly natural Fo contours, the VC system can effectively smooth and refine these contours, resulting in more natural-sounding converted speech.
  • Trade-off between Intelligibility and Naturalness: The results suggest that while the proposed system might slightly degrade intelligibility compared to the original EL speech, it significantly boosts naturalness and preference, indicating a successful trade-off for improved voice quality.

Key Results

  • Objective Results:
    • The correlation coefficient of Fo contours between converted and target speech improved from 0.58 (for standard EL speech) to 0.78 (for EL(air) speech).
  • Unvoiced/voiced (U/V) decision error rate for Fo estimation was reduced to 4.73% for EL(air) speech, compared to 5.44% for standard EL speech.
  • Mel-cepstral distortion was lower for EL(air) speech conversion (9.51) compared to EL speech (8.96), but both were significantly better than EL(small) speech conversion.
  • Subjective Results:
    • Naturalness: Converted EL(air) speech was rated significantly higher in naturalness than converted standard EL speech.
  • Preference: Subjects preferred the converted EL(air) speech over the converted standard EL speech.
  • Intelligibility: While original EL speech had the highest intelligibility, the converted EL and EL(air) speech intelligibility scores were deemed acceptable (around 3.5 on a 5-point scale).

Limitations and Future Work

  • Intelligibility Degradation: The paper acknowledges that the intelligibility of the converted speech is lower than the original EL speech. Improving intelligibility in the VC process is identified as a future work direction.
  • Fo Contour Naturalness: Despite improvements, the paper notes that EL(air) speech contours can still vary discontinuously, and further work is needed to achieve perfectly smooth and natural Fo contours.
  • Training Time: The laryngectomized speaker required significant training (seven weeks) to control the air-pressure sensor effectively. Future work could explore methods to reduce this training burden.
  • Generalizability: The experiments were conducted with a single laryngectomized speaker. Further evaluation with a larger and more diverse group of users would be beneficial.

Summary updated

BibTeX
@inproceedings{nakamura2010use,
  author = {Keigo Nakamura and Tomoki Toda and Hiroshi Saruwatari and Kiyohiro Shikano},
  title = {{The use of air-pressure sensor in electrolaryngeal speech enhancement based on statistical voice conversion}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure