A comparison of speech synthesis systems based on GPR, HMM, and DNN with a small amount of training data

Tomoki Koriyama, Takao KobayashiInterspeech 2015

TL;DR

This paper compares Gaussian Process Regression (GPR), Hidden Markov Model (HMM), and Deep Neural Network (DNN) based speech synthesis systems, particularly when trained on limited data. They found that GPR-based synthesis outperforms both HMM and DNN on spectral and prosodic features with only 40 minutes of training data, suggesting GPR's potential for data-efficient speech synthesis.

What Problem Does This Paper Address?

Statistical parametric speech synthesis (SPSS) aims to generate natural-sounding speech by modeling acoustic features. While HMM-based systems have been a long-standing approach, they can sometimes produce overly smoothed speech. More recently, deep learning approaches like DNNs have shown promise, especially with large datasets, by offering more flexible context modeling. However, the effectiveness of these advanced methods with limited training data remains an open question. This paper investigates an alternative approach, GPR-based synthesis, and directly compares its performance against HMM and DNN under data-scarce conditions, specifically exploring how different levels of linguistic context clustering impact GPR's effectiveness.

Methods

The paper compares three main speech synthesis paradigms:

  • HMM-based Speech Synthesis: This traditional method uses Hidden Markov Models (HMMs) to model acoustic feature distributions within states. Decision trees are employed to map linguistic contexts to these states, and speech parameters are predicted based on the state distributions.
  • DNN-based Speech Synthesis: This approach leverages deep neural networks to directly predict frame-level acoustic features from linguistic contexts. DNNs can model complex relationships between contexts more flexibly than decision trees.
  • GPR-based Speech Synthesis: This method models the relationship between linguistic features and acoustic features using Gaussian Process Regression (GPR). GPR uses kernel functions to define similarities between frames, allowing for non-parametric regression. The paper utilizes a Partially Independent Conditional (PIC) approximation for computational efficiency, which involves clustering data and using inducing points. The paper also investigates different linguistic clustering levels (state, phone, mora, accent phrase) for the GPR system.

Datasets

The experiments were conducted using the ATR Japanese speech database set B.

  • Training Data: 450 sentences per speaker. The total training data for the four speakers (two female, two male) ranged from 36.1 to 41.7 minutes.
  • Test Data: The remaining 53 sentences per speaker.

No new benchmarks were introduced; standard evaluation metrics were used.

Key Insights and Ideas

  • GPR's Data Efficiency: The most striking finding is GPR's superior performance over both HMM and DNN when trained on a small dataset (around 40 minutes). This suggests GPR's kernel-based approach is more robust to limited data for capturing spectral and prosodic nuances.
  • Clustering Matters for GPR: The choice of linguistic context clustering level significantly impacts GPR's performance. For phone duration, phone-level clustering was best, while for mel-cepstrum, log F0, and aperiodicity, mora-level clustering yielded the lowest errors. This highlights the importance of aligning the clustering granularity with the feature being modeled.
  • Mora as a Suitable Unit for Japanese: The study suggests that mora is a well-suited unit for Japanese speech synthesis, balancing context capture with sufficient segment length for trajectory modeling.
  • DNN vs. HMM: As expected, DNNs generally outperform HMMs across most features, even with limited data, demonstrating the power of deep architectures for context modeling.
  • GPR vs. DNN (with limited data): While DNNs are strong, GPR with appropriate clustering can surpass DNNs in spectral and prosodic accuracy when data is scarce.

Key Results

  • Objective Evaluation:
    • GPR with mora-level clustering consistently outperformed DNN (3-layer) for mel-cepstrum, log F0, and aperiodicity models.
  • DNN (3-layer) and GPR with phone clustering were comparable in duration prediction and voiced/unvoiced flag prediction.
  • GPR generally showed lower RMS errors for phone duration and log F0 compared to DNN.
  • Subjective Evaluation:
    • Both DNN and GPR significantly outperformed HMM in naturalness.
  • GPR achieved a slightly higher score than DNN in paired comparison tests, indicating better perceived naturalness under the limited data condition.

Limitations and Future Work

  • Limited Data Scope: The study focused on a specific, relatively small amount of training data (around 40 minutes). The performance of GPR, DNN, and HMM might differ with larger datasets.
  • Specific Database: The experiments were conducted on a specific Japanese speech database. Generalizability to other languages or acoustic conditions is not directly addressed.
  • Future Work: The authors explicitly state their intention to evaluate the performance of these methods with various amounts and kinds of training data in future work. They also suggest exploring dynamic features in GPR, similar to HMM and DNN.

Summary updated

BibTeX
@inproceedings{koriyama2015comparison,
  author = {Tomoki Koriyama and Takao Kobayashi},
  title = {{A comparison of speech synthesis systems based on GPR, HMM, and DNN with a small amount of training data}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure