Learning Robust Latent Representations for Controllable Speech Synthesis

Shakti Kumar, Jithin Pradeep, Hussain ZaidiACL 2021

TL;DR

This paper introduces REMMI, a novel approach to learning disentangled latent representations for speech synthesis. By minimizing mutual information between latent variables and reordering Transformer layers, REMMI achieves more robust control over speech attributes like speaking rate, pitch, and pause duration, especially on limited or noisy datasets where previous methods fail.

What Problem Does This Paper Address?

Controllable Text-to-Speech (TTS) systems aim to synthesize speech with specific stylistic attributes (e.g., pitch, speed, accent). Variational Autoencoders (VAEs) have shown promise in learning these disentangled latent representations. However, existing LSTM-based VAEs struggle to learn distinct clusters for speaker attributes when trained on small or noisy datasets. Furthermore, different latent variables often encode the same features, limiting precise control over synthesis. This paper addresses the challenge of learning robust, disentangled latent representations that enable fine-grained control over speech synthesis, particularly under data-scarce and noisy conditions.

Methods

The authors propose REMMI (Reordered Transformer Encoder with Minimal Mutual Information), which has two main components:

  • Minimizing Mutual Information (LMI): They introduce a novel loss term that explicitly minimizes the mutual information between latent variables (specifically, between the unsupervised latent $z_1$ and the conditional latent $z_0$, which is correlated with the speaker label $y_0$). This penalizes shared features between latents, encouraging them to learn distinct aspects of speech.
  • Layer Reordering in Transformer: They replace the standard LSTM encoder with a modified Transformer architecture. The key innovation is reordering the layers, specifically placing Layer Normalization before the Multi-Head Attention and FeedForward modules. This "layer reordering" aims to establish a direct path for learning a general speech representation early in training, independent of speaker identity, before learning style-specific deviations. They also incorporate GRU-type gating and Leaky ReLU activations to stabilize training and improve cluster separation.

Datasets

The models were trained and evaluated on two datasets:

  • MAILABS: A studio-quality dataset with 35 hours of UK and 39 hours of US speech from 4 professional speakers.
  • Common Voice: A crowd-sourced dataset with 4 hours of UK and 19 hours of US speech from 477 volunteers, characterized by varying background noise, recording quality, and a large number of speakers.

Key Insights and Ideas

  • Data Scarcity is a Major Hurdle: State-of-the-art VAEs for speech synthesis collapse or fail to learn meaningful separations of speaking styles when presented with limited datasets.
  • Latent Variable Overlap is Detrimental: When different latent variables encode the same features, it hinders precise control over speech synthesis attributes.
  • Mutual Information Minimization is Key: Explicitly penalizing mutual information between latent variables is crucial for disentangling features and improving control.
  • Transformer Architecture Benefits from Reordering: Reordering Transformer layers (specifically, placing LayerNorm earlier) helps in learning a general speech representation before style-specific features, leading to better disentanglement and faster convergence.
  • Leaky ReLU for Better Cluster Separation: The choice of activation function (Leaky ReLU with $\alpha=0.05$) proved crucial for effectively separating latent clusters corresponding to different speaker attributes.

Key Results

  • Improved Cluster Separation: REMMI reduces speaker attribute cluster overlap by at least 30% compared to LSTM-VAE on limited MAILABS datasets and by half on limited Common Voice datasets.
  • Better Control on Noisy/Small Datasets: REMMI consistently outperforms Transformer-VAE and LSTM-VAE on both MAILABS and Common Voice datasets, showing significantly better DI and DBI scores (indicating better cluster quality).
  • Faster Convergence: REMMI demonstrates accelerated convergence of the variational lower bound compared to both Transformer-VAE and LSTM-VAE.
  • Controllable Synthesis: REMMI allows for precise control over speaking rate, pause duration, and fundamental frequency (pitch) by manipulating latent variables. Human evaluations (CMOS and CS) show REMMI produces speech that sounds more British and offers better control over synthesized accent compared to LSTM-VAE.

Limitations and Future Work

  • Limitations: The paper acknowledges that while REMMI improves control, there's still room for improvement in disentangling all latent variables. The performance gap between REMMI and LSTM-VAE increases as dataset size decreases, suggesting that even REMMI benefits from more data.
  • Future Work: The authors suggest exploring the application of REMMI beyond speech synthesis, such as in image captioning with sentiments or text-to-image rendering with different emotions. They also mention investigating the specific choice of Leaky ReLU and the importance of gates in their ablation study.

Summary updated

BibTeX
@inproceedings{kumar2021learning,
  author = {Shakti Kumar and Jithin Pradeep and Hussain Zaidi},
  title = {{Learning Robust Latent Representations for Controllable Speech Synthesis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure