Speech Quality Evaluation of Synthesized Japanese Speech Using EEG

Ivan Halim Parmonangan, Hiroki Tanaka, Sakriani Sakti, Shinnosuke Takamichi, Satoshi NakamuraInterspeech 2019

TL;DR

This paper explores using electroencephalography (EEG) to objectively evaluate the quality of synthesized Japanese speech. By analyzing brain activity while listeners hear different speech types, the researchers demonstrate that EEG signals can be used to classify speech quality with high accuracy and even predict subjective MOS scores, offering a more detailed understanding than traditional methods.

What Problem Does This Paper Address?

Synthesized speech is becoming ubiquitous, but ensuring its quality remains a challenge. Traditional subjective metrics like Mean Opinion Score (MOS) provide a general impression but lack granular detail about why a speech sample is perceived as good or bad. Objective acoustic metrics, while useful, don't always perfectly correlate with human perception. This gap leaves a need for methods that can objectively assess synthesized speech quality with a deeper understanding of the listener's experience. This paper aims to bridge this gap by leveraging the high temporal resolution and objective nature of EEG signals to predict perceived speech quality.

Methods

The core of this research lies in using EEG to capture listeners' neural responses to synthesized speech. The process involved:

  • Speech Stimuli: A set of 53 Japanese sentences were used, including one natural speech recording and four types of synthesized speech. These synthesized types varied in which speech parameters (fundamental frequency - LF0, mel-cepstrum - MCC, or both) were generated by a deep neural network (DNN) and which were taken from the natural speech.
  • EEG Data Collection: Ten subjects listened to these speech samples while their EEG was recorded using 32 scalp channels. During the listening, participants were instructed to press a button whenever they heard parts of the synthesized speech that sounded "bad."
  • EEG Pre-processing: Raw EEG data was re-referenced, band-pass filtered (0.5-40 Hz), and down-sampled. Epochs corresponding to speech segments were extracted, and artifacts were removed.
  • Feature Extraction: The pre-processed EEG epochs were transformed into the time-frequency domain using Morlet wavelet transform and then divided into four frequency bands: theta (4-8 Hz), alpha (8-13 Hz), beta (13-31 Hz), and gamma (31-38 Hz). This resulted in four distinct datasets, each representing a different frequency band.
  • Classification: Machine learning models (Linear Discriminant Analysis - LDA, L1-regularized Logistic Regression - LR, and Support Vector Machine - SVM) were trained to classify between natural speech and each of the four synthesized speech types. This was performed for each frequency band separately.
  • Regression: To predict subjective quality, regression models (Linear Regression and Support Vector Regression - SVR) were used to predict the actual MOS scores from the EEG features, again, for each frequency band.

Datasets

The study utilized a custom dataset comprising:

  • 53 single-speaker Japanese sentences.
  • One natural speech recording of these sentences.
  • Four types of synthesized speech generated using a DNN-based approach, varying in the generation of LF0 and MCC parameters.
  • EEG recordings from 10 subjects listening to these speech samples.
  • MOS scores collected from 10 subjects for subjective evaluation.

The paper does not introduce a new benchmark dataset but rather uses its collected data for evaluation.

Key Insights and Ideas

  • EEG as a Proxy for Perceived Quality: The most significant insight is that EEG signals, which capture brain activity, can serve as a reliable objective measure for perceived speech quality, going beyond simple acoustic features.
  • Temporal Resolution Matters: The high temporal resolution of EEG allows for capturing subtle neural responses to speech, which is crucial for understanding listener reactions to short speech segments.
  • Frequency Bands Hold Clues: Different frequency bands within the EEG spectrum appear to be more informative for specific comparisons. The alpha band (8-13 Hz) showed particular promise for classification tasks.
  • SVM Outperforms for Classification: Support Vector Machines (SVM) generally achieved the highest classification accuracy, suggesting their effectiveness in distinguishing between different speech quality levels.
  • SVR Offers Better Regression: For predicting MOS scores, Support Vector Regression (SVR) consistently yielded lower Root Mean Squared Error (RMSE) compared to linear regression, indicating its superiority in capturing the nuances of subjective quality.

Key Results

  • Classification Accuracy: The study achieved a maximum classification accuracy of 96.61% using SVM for the "Natural vs. Syn LFO" comparison in the beta band. Generally, accuracies above 79% were observed for classification tasks.
  • Regression Performance: The lowest RMSE achieved for predicting MOS scores was 1.133 using SVR, demonstrating a strong ability to predict subjective quality.
  • Alpha Band Dominance: The alpha frequency band (8-13 Hz) was found to be generally the most effective for differentiating between natural and synthesized speech in classification tasks.
  • SVR Superiority: SVR consistently outperformed linear regression in predicting MOS scores across all frequency bands.

Limitations and Future Work

  • Cross-Subject Generalization: The study aimed for cross-subject generalization, which might lead to slightly lower prediction performance compared to within-subject analysis (as done in some previous work).
  • Limited Speech Material: The study used a relatively small set of 53 sentences.
  • Future Work: The authors suggest exploring within-subject evaluation for potentially higher accuracy. They also propose investigating more sophisticated algorithms and higher-dimensional data representations, such as tensor representations, to further enhance prediction capabilities.

Summary updated

BibTeX
@inproceedings{parmonangan2019speech,
  author = {Ivan Halim Parmonangan and Hiroki Tanaka and Sakriani Sakti and Shinnosuke Takamichi and Satoshi Nakamura},
  title = {{Speech Quality Evaluation of Synthesized Japanese Speech Using EEG}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure