EPG2S: Speech Generation and Speech Enhancement Based on Electropalatography and Audio Signals Using Multimodal Learning

Li‐Chin Chen, P. s. Chen, Richard Tzong‐Han Tsai, Yu TsaoIEEE Signal Processing Letters 2022

TL;DR

This paper introduces EPG2S, a novel multimodal system that generates and enhances speech using electropalatography (EPG) and audio signals. By combining these modalities, EPG2S offers a promising avenue for restoring verbal communication for individuals with speech impairments and improving speech intelligibility in noisy environments.

What Problem Does This Paper Address?

For individuals who have lost the ability to speak due to conditions like vocal cord damage, verbal communication becomes a significant challenge. While alternative communication methods exist, speaking remains the most natural and familiar form of interaction. This paper addresses the need for assistive technologies that can enable verbal communication by translating articulatory movements into speech. Furthermore, the increasing prevalence of mask-wearing due to public health concerns can obscure oral movements, hindering communication. The paper also tackles the problem of speech enhancement in noisy environments, a common issue that degrades the quality and intelligibility of spoken language.

Methods

The core of the EPG2S system is a multimodal learning framework that leverages both Electropalatography (EPG) and audio speech signals. EPG is a non-invasive technique that records the contact patterns between the tongue and the hard palate during speech, offering a robust signal unaffected by background noise. The EPG2S system comprises two main components: an EPG-to-speech (E2S) component for generating speech and a spectrogram feature extraction (SFE) component for processing audio.

The E2S component includes an Encoder (Encodere) that converts EPG signals into a latent representation and a Decoder that reconstructs speech spectrograms. The SFE component uses a Convolutional Neural Network (CNN)-based Encoder (Encodera) to extract features from audio spectrograms.

The paper explores two primary fusion strategies for combining EPG and audio signals:

  • Early Fusion (EF): This strategy concatenates the EPG and audio features at an early stage before feeding them into the model. To handle the heterogeneous nature of the data, the EPG signals are linearly transformed to match the dimensionality of the audio features.
  • Late Fusion (LF): In this approach, EPG and audio signals are processed independently by their respective encoders to extract latent representations. These representations are then concatenated and fed into a fusion layer, followed by the decoder.

The system is trained to minimize both the spectral reconstruction loss (mean squared error between the generated and target spectrograms) and a "join" loss that encourages the latent representations from EPG and audio to be similar.

Datasets

The study utilized a dataset collected using the CompleteSpeech SmartPalate® system. EPG signals and corresponding audio speech were recorded simultaneously from a male speaker reading the Taiwan Mandarin Hearing in Noise Test (TMHINT) corpus, which consists of 320 sentences. After excluding one utterance due to poor quality, 319 pairs of EPG and speech signals were used. The dataset was split into training (222 utterances), validation (27 utterances), and testing (70 utterances) sets. For speech enhancement experiments, artificial noise (vehicular, engine, street, and background speaker noise) was added to clean speech at various Signal-to-Noise Ratios (SNRs) ranging from -10 dB to 10 dB.

Key Insights and Ideas

  • EPG as a viable source for speech generation: The research demonstrates that EPG signals alone can produce desirable speech generation outcomes, highlighting its potential for individuals unable to produce vocal speech.
  • Synergy between EPG and audio: Combining EPG with noisy speech signals, even during training, significantly improves the quality and intelligibility of the generated speech. This suggests that EPG can compensate for some of the information lost in noisy audio.
  • EPG's role in speech enhancement: EPG signals act as a beneficial auxiliary input for enhancing noisy speech, leading to improved performance across various metrics, even when the audio signal is already distorted.
  • Late Fusion superiority: The Late Fusion (LF) strategy emerged as the most effective approach for both speech generation and enhancement, outperforming Early Fusion (EF) in several key metrics. This suggests that processing modalities independently before combining them allows for better feature extraction and integration.
  • Personalized system effectiveness: The study successfully constructed a personalized EPG2S system, indicating that the approach is adaptable to individual articulatory patterns.

Key Results

  • Speech Generation:
    • Pure EPG signals yielded desirable speech generation outcomes.
  • The addition of noisy speech signals (via LF) improved quality and intelligibility compared to using pure EPG.
  • LF outperformed pure EPG in PESQ, ESTOI, and MCD (noisy), while EF showed an advantage in STOI.
  • Speech Enhancement:
    • EPG2S demonstrated high-quality speech enhancement based on audio signals alone.
  • The addition of EPG signals further improved performance across various metrics (PESQ, STOI, ESTOI, MCD) compared to the baseline (audio-only enhancement).
  • LF generally performed better than EF in speech enhancement, particularly in terms of MCD and SSNR.

Limitations and Future Work

  • Personalization Scope: While the study successfully demonstrated a personalized system, the authors acknowledge that constructing personalized models for a larger number of individuals needs further verification.
  • Data Size: The dataset consisted of signals from a single speaker. Future work should explore the scalability and generalizability of the EPG2S system across a diverse range of speakers.
  • Real-world Deployment: The paper suggests that while audio recording is optional, its availability is beneficial. Further research could investigate the optimal balance and integration of EPG and audio in real-world assistive communication devices.
  • Fusion Strategy Optimization: While LF was found to be superior, further exploration of advanced fusion techniques and hyperparameter tuning could potentially yield even better results.

Summary updated

Abstract

Speech generation and enhancement based on articulatory movements facilitate communication when the scope of verbal communication is absent, e.g., in patients who have lost the ability to speak. Although various techniques have been proposed to this end, electropalatography (EPG), which is a monitoring technique that records contact between the tongue and hard palate during speech, has not been adequately explored. Herein, we propose a novel multimodal EPG-to-speech (EPG2S) system that utilizes EPG and speech signals for speech generation and enhancement. Different fusion strategies based on multiple combinations of EPG and noisy speech signals are examined, and the viability of the proposed method is investigated. Experimental results indicate that EPG2S achieves desirable speech generation outcomes based solely on EPG signals. Further, the addition of noisy speech signals is observed to improve quality and intelligibility. Additionally, EPG2S is observed to achieve high-quality speech enhancement based solely on audio signals, with the addition of EPG signals further improving the performance. The late fusion strategy is deemed to be the most effective approach for simultaneous speech generation and enhancement.
BibTeX
@article{chen2022epg,
  author = {Li‐Chin Chen and P. s. Chen and Richard Tzong‐Han Tsai and Yu Tsao},
  title = {{EPG2S: Speech Generation and Speech Enhancement Based on Electropalatography and Audio Signals Using Multimodal Learning}},
  journal = {IEEE Signal Processing Letters},
  year = {2022},
}
Figure