Permanent Magnetic Articulograph (PMA) vs Electromagnetic Articulograph (EMA) in Articulation-to-Speech Synthesis for Silent Speech Interface

Beiming Cao, Nordine Sebkhi, Ted Mau, Omer T. Inan, Jun WangNAACL 2019

TL;DR

This paper compares two technologies for silent speech interfaces (SSIs): the established Electromagnetic Articulograph (EMA) and a newer, wireless Permanent Magnetic Articulograph (PMA). The study finds that while EMA still offers superior performance in converting articulatory movements to speech, PMA shows comparable results using just tongue tip data and holds significant promise for practical SSI applications due to its wireless and affordable nature.

What Problem Does This Paper Address?

For individuals who have undergone a laryngectomy (e.g., due to laryngeal cancer), the ability to produce audible speech is lost. Existing communication aids, such as artificial larynxes or esophageal speech, often result in abnormal or unnatural-sounding speech, leading to social anxiety and reduced quality of life. Silent Speech Interfaces (SSIs) offer a promising solution by converting non-audible biological signals, like articulatory movements, into synthesized speech. A key challenge in SSI development is accurately capturing these articulatory movements and effectively translating them into natural-sounding speech. This paper specifically addresses the performance comparison between two leading technologies for capturing these movements: the commercially available, wired Electromagnetic Articulograph (EMA) and the emerging, wireless Permanent Magnetic Articulograph (PMA), in the context of Articulation-to-Speech (ATS) synthesis.

Methods

The core of this research lies in comparing the ATS performance of PMA and EMA. The authors employed a Deep Neural Network (DNN) architecture for the ATS synthesis, mapping articulatory signals to acoustic features (mel-cepstral coefficients or MCCs). The DNN comprised 6 hidden layers with 512 nodes each, utilizing a rectified linear unit (ReLU) activation function and the Adam optimizer.

Two key experimental setups were used:

  1. PMA Data Analysis: The researchers investigated whether using raw magnetic field signals from PMA, converted positional data (3D coordinates), or a combination of both yielded better ATS performance. This experiment was specific to their newly developed PMA system.
  2. PMA vs. EMA Comparison: The best-performing PMA data configuration was then directly compared against the EMA data.

To ensure a fair comparison, the audio data from both PMA and EMA were resampled to 48000 Hz, and spectral envelopes were extracted to generate 60-dimensional mel-cepstral coefficients (MCCs) at a rate of 200 frames per second. The PMA and EMA data were upsampled to match this acoustic feature rate.

Datasets

The study utilized datasets collected from human subjects speaking a list of 132 phrases.

  • PMA Dataset: Collected from 10 subjects (6 male, 4 female) at the Georgia Institute of Technology. Each subject repeated the 132 phrases twice (once voiced, once unvoiced). For this study, only the voiced speech data was used.
  • EMA Dataset: Collected from 10 age- and gender-matched subjects (6 male, 4 female) at the University of Texas at Dallas. These subjects read the same list of 132 phrases.

For both datasets, 110 phrases were used for training, 10 for validation, and 12 for testing.

Key Insights and Ideas

  • PMA's Versatility: The study revealed that for PMA, using raw magnetic signals, converted positional data, or a combination of both resulted in statistically similar ATS performance. This is a significant finding because it suggests flexibility in how PMA data can be processed for ATS, and importantly, positional data opens doors to established articulation processing methods (like Procrustes matching) and simplifies device configuration.
  • Wireless Advantage: The wireless nature of PMA, coupled with its affordability and portability, makes it a highly attractive alternative to the more cumbersome and expensive EMA for practical SSI applications, even if its current performance is slightly lower.
  • Tongue Tip Sufficiency: The research demonstrated that using only tongue tip data from PMA could achieve comparable ATS performance to using combined raw and positional data. This simplifies the data acquisition process and reduces the complexity of the PMA system.

Key Results

  • PMA Data Configurations: No significant difference was found in ATS performance between using raw PMA magnet signals, positional data, or both.
  • PMA vs. EMA: The EMA-based ATS system significantly outperformed all PMA configurations (p < 0.01). However, the PMA performance was still described as "close to" EMA's, indicating a strong potential.
  • Mel-Cepstral Distortion (MCD): Average MCD values were reported, with EMA achieving an average of 6.87, while PMA (using raw, positional, and both) ranged from 7.74 to 7.88. Lower MCD values indicate better speech synthesis quality.

Limitations and Future Work

  • Lip Movement Exclusion: The study did not incorporate lip movement information, as the PMA and EMA devices used different approaches for capturing it (PMA used computer vision, EMA relied on sensor tracking). Future work could integrate these.
  • Subjectivity of Performance Measure: The primary performance measure was Mel-Cepstral Distortion (MCD). While widely used, it doesn't fully capture vocal quality. The authors suggest that other acoustic measures and listening tests are needed for a more comprehensive assessment.
  • Data Size and Subject Variation: The study acknowledges that the relatively small dataset size and the use of different subjects for PMA and EMA data collection (due to different research labs) could introduce variations.
  • Future Directions: The authors propose future work to:
  • Verify findings with larger datasets from the same speakers using both EMA and PMA.
  • Improve PMA measurement accuracy and the localization approach for converting raw signals to positional data.
  • Explore the integration of lip movement data.
  • Conduct listening tests to assess subjective speech quality.

Summary updated

BibTeX
@inproceedings{cao2019permanent,
  author = {Beiming Cao and Nordine Sebkhi and Ted Mau and Omer T. Inan and Jun Wang},
  title = {{Permanent Magnetic Articulograph (PMA) vs Electromagnetic Articulograph (EMA) in Articulation-to-Speech Synthesis for Silent Speech Interface}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure