Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis Through Audio Analysis

Noé Tits, Fengna Wang, Kevin El Haddad, Vincent Pagel, Thierry DutoitInterspeech 2019

TL;DR

This paper tackles the challenge of controlling expressive speech synthesis by analyzing and visualizing the "latent spaces" learned by deep learning models. By comparing different methods of creating these spaces, the authors demonstrate that they can capture meaningful variations in speech, paving the way for more intuitive and controllable expressive TTS systems.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems, especially those powered by deep learning, have achieved remarkable realism. However, a key limitation is the difficulty in controlling the expressiveness of the synthesized speech – making it sound happy, sad, angry, or even whispery. While deep learning models learn internal representations (latent spaces) that implicitly capture these expressive nuances, these spaces are often complex and difficult to interpret. Researchers struggle to understand what these latent variables represent and how they can be manipulated to achieve desired expressive outcomes. This paper aims to bridge this gap by analyzing and visualizing these latent spaces, making them more understandable and usable for controlling expressive TTS.

Methods

The core of this work involves training three different deep learning systems to generate latent space embeddings from speech data. The goal is to see which type of embedding best captures expressive speech characteristics.

  1. Style Classification System: A Long Short-Term Memory (LSTM) based neural network is trained to classify speech utterances into one of eight predefined emotional/stylistic categories (e.g., happy, sad, neutral). The output of a specific layer in this network serves as the style embedding.
  2. Speaker Classification System: Similar to the style classifier, this LSTM-based network is trained to predict the identity of the speaker. While this system doesn't explicitly use style labels during training, the authors hypothesize that speaker embeddings might implicitly capture some expressive information due to variations in speaking style across different speakers.
  3. VAE-TTS System: This approach uses a Variational Auto-Encoder (VAE) integrated into a Text-to-Speech (TTS) architecture (specifically, a variant of DCTTS). The VAE is trained to encode mel-spectrograms (a representation of audio) into a latent vector. This vector is then used to condition the TTS model, allowing it to generate speech with variations not directly dictated by the text. Crucially, this system is trained without explicit style labels.

To understand and compare these latent spaces, the researchers employ:

  • Mutual Information: To quantify how much information each dimension of the latent space shares with the predefined style categories.
  • Linear Regression: To approximate relationships between the latent space dimensions and various audio features (like pitch, formants, and spectral characteristics) using the eGeMAPS feature set. The accuracy of these approximations is measured by the Absolute Pearson Correlation Coefficient (APCC).
  • Dimensionality Reduction (PCA, t-SNE, UMAP): To project the high-dimensional latent spaces into 2D, enabling visualization and the identification of trends related to audio features.

Datasets

The primary dataset used is a proprietary dataset from Acapela Group SA. This dataset consists of phonetically rich sentences recorded by a single male actor ("Will") in eight distinct style classes: neutral, happy, sad, bad guy, from afar, proxy (whispering), old man, and little creature. The dataset includes audio files and corresponding transcriptions.

Key Insights and Ideas

  • Latent Spaces Capture Expressive Nuances: Even when not explicitly trained for style classification (like the speaker classification and VAE-TTS systems), the learned latent spaces show a remarkable ability to group utterances by style. This suggests that these internal representations are inherently sensitive to expressive variations.
  • Style Classification Embeddings are Most Discriminative: As expected, embeddings from the style classification system yield the highest mutual information with style categories, indicating they are most directly tuned to represent expressive styles.
  • Audio Feature Correlations Reveal Control Potential: By correlating latent space dimensions with specific audio features (like F0 mean, MFCCs, and formant frequencies), the authors identify which acoustic characteristics are most strongly influenced by the latent variables. This is crucial for understanding how to manipulate the latent space to control specific aspects of speech expressiveness.
  • Visualization Aids Interpretation: Reducing the latent spaces to 2D allows for visual exploration of how different styles and audio features are distributed. This provides an intuitive way to grasp the structure of the learned representations.

Key Results

  • The style classification system achieved 94.38% accuracy on its evaluation set.
  • The speaker classification system achieved 91.19% classification accuracy.
  • Table 3 shows that the VAE-TTS latent space generally exhibits strong correlations (APCC > 0.5) with many audio features, particularly for features like "mfcc4 mean" (0.73), "mfcc4 V mean" (0.89), and "mfcc2 V mean" (0.78), suggesting its potential for controlling these aspects.
  • The Style latent space also shows strong correlations, often exceeding those of the VAE-TTS system for features like "F0 mean" (0.82) and "F0 percentile 50.0" (0.86).
  • The Speaker latent space shows moderate correlations, generally lower than the other two, but still significant for some features.
  • Table 4 indicates that when dimensionality reduction algorithms (PCA, t-SNE, UMAP) are applied, the VAE-TTS and Speaker latent spaces tend to retain more information (higher average APCC) compared to the Style latent space, especially when using PCA.

Limitations and Future Work

  • Limited Dataset: The study relies on a single actor's voice and a specific set of styles. The generalizability of these findings to other speakers, languages, and a wider range of expressive nuances remains to be explored.
  • Linear Approximation: The analysis of audio features relies on linear regression. More complex, non-linear relationships might exist between latent spaces and audio features, which could be captured by more advanced modeling techniques.
  • Future Work: The authors plan to use the developed visualization techniques and understanding of latent space-audio feature relationships to build an interface for controlling expressive speech synthesis systems. They also aim to explore more complex latent space structures and their impact on speech quality and expressiveness.

Summary updated

BibTeX
@inproceedings{tits2019visualization,
  author = {Noé Tits and Fengna Wang and Kevin El Haddad and Vincent Pagel and Thierry Dutoit},
  title = {{Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis Through Audio Analysis}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure