MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis

Wenhao Guan, Yishuang Li, Tao Li, Hukai Huang, Feng Wang, Jiayan Lin, Lingyan Huang, Lin Li, Qingyang HongAAAI 2024 · DOI

TL;DR

This paper introduces MM-TTS, a novel text-to-speech (TTS) system that allows for flexible style control using prompts from various modalities like speech, images, and text descriptions. It addresses the limitations of existing TTS systems that rely on fixed emotional labels or single-modality references by unifying these diverse prompts into a single style space.

What Problem Does This Paper Address?

Traditional Text-to-Speech (TTS) systems have made significant strides in generating natural-sounding speech. However, a key challenge remains in controlling the expressiveness of synthesized speech. Current methods for style transfer in TTS often rely on predefined emotional labels or require a reference speech clip that matches the target style. This approach is rigid and limits the flexibility of style control. While some recent work has explored using text descriptions for style guidance, the field lacks a unified framework that can seamlessly integrate and leverage multiple modalities (speech, images, text) to control speech style. This paper aims to bridge this gap by proposing a multi-modal prompt-based style transfer framework for expressive TTS.

Methods

The core of MM-TTS lies in its ability to unify diverse modal inputs into a single "style space." This is achieved through several novel components:

  • Aligned Multi-Modal Prompt Encoder (AMPE): This module uses the power of CLIP (Contrastive Language-Image Pretraining) to align text, speech, and image prompts into a unified style representation. It leverages CLIP's text and image encoders, along with a dedicated speech style encoder, and uses adapter modules to map these into a common space.
  • Style Adaptive Convolutions (SAConv): To effectively transfer the unified style representation to the target text content, MM-TTS introduces SAConv. This module predicts adaptive convolution kernels and biases, allowing for more precise style transfer compared to methods that simply concatenate or sum style features.
  • Rectified Flow Based Refiner: To combat the common issue of over-smoothing in Mel-spectrograms generated by TTS models, a Rectified Flow based refiner is employed. This module uses an Ordinary Differential Equation (ODE) solver to refine Mel-spectrograms, producing outputs that are closer to the real data distribution and improving audio fidelity.
  • Two-Stage Training: The system is trained in two stages: first, a text-to-Mel model is trained, and then the Rectified Flow Refiner is trained to further enhance the Mel-spectrograms.

Datasets

  • MEAD-TTS: The authors introduce a new multi-modal expressive TTS dataset called MEAD-TTS. This dataset is constructed from the MEAD dataset (an expressive talking head video dataset) and includes text transcriptions, face images, and speech prompts. It aims to provide a benchmark for multi-modal TTS research.
  • LJSpeech: Used for pretraining the model to increase the number of text transcription data and enhance TTS output diversity.
  • LibriTTS: Used for out-of-domain evaluation of reference speech-based style transfer.
  • Oulu-CASIA: Used for out-of-domain evaluation of face-based style transfer.

Key Insights and Ideas

  • Unified Style Space: The most significant insight is the successful unification of diverse modal prompts (speech, image, text) into a single style space. This allows for unprecedented flexibility in controlling speech style.
  • CLIP for Cross-Modal Alignment: Leveraging CLIP's pre-trained capabilities for aligning text and image modalities is a clever way to establish a common ground for style representation.
  • Adaptive Convolutions for Precision: The SAConv module's ability to predict adaptive kernels and biases offers a more granular control over style transfer compared to global style feature injection.
  • Rectified Flow for Fidelity: The application of Rectified Flow for Mel-spectrogram refinement is a promising approach to address the over-smoothing issue and improve the realism of synthesized audio.

Key Results

  • Reference Speech Style Transfer: MM-TTS achieved superior results in both parallel and non-parallel style transfer tasks, outperforming baseline models in terms of MOS (Mean Opinion Score) and SMOS (Similarity Mean Opinion Score). For example, in intra-domain PST, MM-TTS achieved a MOS of 4.36 ± 0.06 and SMOS of 4.61 ± 0.07.
  • Face Based Style Transfer: The model demonstrated effectiveness in capturing and transferring style attributes from facial images, leading to better audio naturalness and classification accuracy for emotion and gender.
  • Text Description Based Style Transfer: MM-TTS also showed strong performance in modeling and incorporating styles from natural language descriptions, achieving superior MOS and classification accuracy.
  • Ablation Studies: The results from ablation studies confirmed the importance of each proposed component: SAConv significantly improved style similarity, the Reflow Refiner enhanced audio naturalness, and pretraining on LJSpeech was crucial for out-of-domain performance.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the newly constructed MEAD-TTS dataset is relatively small, which limits its applicability to real-world scenarios, especially for out-of-domain evaluations.
  • Text Prompt Template Limitations: The devised text prompt template has limitations in comprehending more abstract or complex styles, restricting the system's ability to capture nuanced stylistic attributes.

Future work could focus on expanding the MEAD-TTS dataset and developing more sophisticated methods for interpreting and generating complex stylistic descriptions. The paper suggests that MM-TTS represents a significant step towards more universal TTS systems with multi-modal capabilities.

Summary updated

BibTeX
@inproceedings{guan2024mm,
  author = {Wenhao Guan and Yishuang Li and Tao Li and Hukai Huang and Feng Wang and Jiayan Lin and Lingyan Huang and Lin Li and Qingyang Hong},
  title = {{MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29769},
}
Figure