Phone-Level Embeddings for Unit Selection Speech Synthesis

Antoine Perquin, Gwénolé Lecorvé, Damien Lolive, Laurent AmsalegSLSP 2018 · DOI

TL;DR

This paper introduces phone-level embeddings for unit selection speech synthesis, enabling a more efficient and adaptable approach by replacing expert-defined cost functions with simple Euclidean distances in an embedding space. Their models, particularly DNN-BP, achieve perceptually preferred synthesized speech over traditional expert-based methods and demonstrate good generalization across different speech styles.

What Problem Does This Paper Address?

Traditional unit selection speech synthesis, while capable of producing natural-sounding speech, relies heavily on meticulously crafted cost functions designed by linguistic experts. This process is time-consuming, expensive, and difficult to adapt to new domains or languages. The paper aims to address this by developing a method to automatically learn representations (embeddings) of speech units that capture relevant linguistic and acoustic information, allowing for simpler, data-driven selection criteria. The challenge lies in creating embeddings that are both informative enough for high-quality synthesis and generalizable across different speech characteristics.

Methods

The authors propose four deep neural network (DNN) architectures to extract phone-level embeddings. Three are feed-forward DNNs (DNN-S, DNN-B, DNN-BP), and one uses a Long Short-Term Memory (LSTM) layer (LSTM-BP).

  • DNN-S: A standard feed-forward DNN that predicts acoustic features for each frame based on linguistic features and timing information.
  • DNN-B: Introduces a "bottleneck" layer, gradually reducing and then increasing the hidden layer sizes. This compresses the representation, potentially mitigating the curse of dimensionality and facilitating similarity measures.
  • DNN-BP: Modifies DNN-B by postponing the use of timing information until after the embedding layer, aiming to create phone-level embeddings rather than frame-level ones.
  • LSTM-BP: Replaces a layer in DNN-BP with an LSTM layer to model temporal dependencies across frames within a phone, trained on sequences rather than shuffled frames.

The core innovation is using these learned embeddings to define the "target cost" in unit selection. Instead of complex expert-defined functions, the cost is simply the Euclidean distance between the embedding of a target phone and the embedding of a candidate unit from the speech database.

Datasets

The models were trained and evaluated on a French audiobook corpus, providing approximately 11 hours of speech from a professional speaker. This corpus is noted for its expressive narration, complex sentences, and formal register, making it a challenging dataset. For evaluation, 105 utterances were held out for listening tests. Out-of-domain utterances were also used for testing generalization.

Key Insights and Ideas

  • Embeddings as a Replacement for Expert Knowledge: The most significant insight is that learned phone-level embeddings can effectively replace intricate, expert-designed cost functions in unit selection, simplifying the process and enabling easier adaptation.
  • Bottleneck Architecture's Effectiveness: The bottleneck architecture (DNN-B and DNN-BP) did not hurt acoustic feature prediction quality and, in fact, showed comparable or slightly better results than the standard DNN-S. This suggests that information compression can be beneficial.
  • Phone-Level vs. Frame-Level Embeddings: The DNN-BP model, by focusing on phone-level embeddings, demonstrated the viability of this approach for unit selection, which inherently operates at the phone level.
  • LSTM's Mixed Results: While LSTM-BP was designed to capture temporal dependencies, it performed poorly in statistical parametric speech synthesis (SPSS) due to training on sequences, which reduced data diversity. However, surprisingly, it performed well in the unit selection paradigm, suggesting that the quality of embeddings is not solely determined by objective acoustic modeling performance.
  • Generalization of Embeddings: The proposed embeddings showed good generalization capabilities, performing well on both in-domain and out-of-domain utterances in unit selection.

Key Results

  • Objective Evaluation: While not always outperforming other models in raw acoustic feature prediction (e.g., LSTM-BP had higher errors), the DNN-B and DNN-BP models showed that the bottleneck approach was effective.
  • Perceptual Evaluation (SPSS): In statistical parametric speech synthesis, the models (DNN-S, DNN-B, DNN-BP) performed significantly worse than a baseline "perfect" SPSS system (VOC), with mean scores around 3 out of 10. LSTM-BP performed very poorly.
  • Perceptual Evaluation (Unit Selection): This is where the embeddings shone.
  • In-domain: System DNN-BP received a mean score of 6.8, and LSTM-BP received 6.6, both significantly outperforming the expert system (mean score 5.4).
  • Out-of-domain: DNN-BP scored 7.0, and LSTM-BP scored 6.4, again outperforming the expert system (mean score 5.9). The difference between DNN-BP and LSTM-BP was significant here.
  • The paper concludes that the embedding systems are general enough to be used in other domains.

Limitations and Future Work

  • Link between Acoustic Modeling and Embedding Quality: The authors acknowledge that the link between the quality of acoustic feature prediction and the quality of the resulting unit embeddings is not yet clear. LSTM-BP's poor SPSS performance but good unit selection results highlight this disconnect.
  • LSTM Performance: The LSTM-BP model's performance was not conclusive, particularly in SPSS. Further research is needed to understand why it performed poorly in one paradigm but well in another.
  • Objective Metrics for Embeddings: There is a lack of objective metrics to evaluate the quality of embeddings for speech synthesis. The authors suggest exploring topological properties of embedding spaces.
  • Future Directions:
    • Investigating multi-modality (combining different types of information) for better embeddings, as suggested by prior work.
  • Integrating duration information directly into the embeddings.
  • Extending the approach to handle large, heterogeneous speech databases (multiple speakers, languages).
  • Developing efficient approximate nearest neighbor search techniques for fast database lookups with embeddings.

Summary updated

BibTeX
@inproceedings{perquin2018phone,
  author = {Antoine Perquin and Gwénolé Lecorvé and Damien Lolive and Laurent Amsaleg},
  title = {{Phone-Level Embeddings for Unit Selection Speech Synthesis}},
  booktitle = {SLSP},
  year = {2018},
  doi = {10.1007/978-3-030-00810-9\_3},
}
Figure