Abstract
Using a text description as a prompt to guide the generation of text or images has drawn wide attention recently. Beyond text and image generation, in this work, we explore the possibility of utilizing text descriptions to guide speech synthesis. Thus, we develop a text-to-speech (TTS) system (dubbed as PromptTTS) that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. Specifically, PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt, and a speech decoder to synthesize speech according to the extracted style and content representations. Experiments show that PromptTTS can generate speech with precise style control and high speech quality.
Introduction
Controllable Text-to-Speech (TTS) has traditionally been an "expert-only" domain. To modify the pitch, speed, or emotion of a synthetic voice, users typically had to manually provide numerical values for acoustic features or find a reference audio clip for the model to mimic. These methods are either technically demanding or time-consuming.
PromptTTS addresses this gap by allowing users to control the "how" of speech—style, emotion, and pacing—using simple natural language descriptions. By bridging the gap between natural language style descriptions and underlying acoustic representations, PromptTTS makes high-quality, controllable speech synthesis accessible to non-experts through intuitive prompts like "A lady whispers to her friend slowly."
Methodology
The PromptTTS architecture is an end-to-end system consisting of three primary modules designed to process semantic and acoustic information simultaneously.
- Style Encoder: This module utilizes a pre-trained BERT model to process the "style prompt." To ensure the model recognizes style-related keywords (e.g., "shout," "whisper"), the authors fine-tuned BERT using P-tuning v2 on an auxiliary task: predicting gender, pitch, speed, volume, and emotion from the text.
- Content Encoder: This takes the "content prompt" (the text to be spoken) and converts it into phonemes. It employs a variance adaptor to predict duration and pitch, conditioned heavily on the style representation extracted by the Style Encoder.
- Speech Decoder: A Transformer-based decoder merges the style and content representations to generate a mel-spectrogram, which is then converted into audible speech using a pre-trained HiFiGAN vocoder.
A key innovation is the end-to-end latent space transfer. By passing a rich semantic vector through the model rather than predicting intermediate numerical values, PromptTTS avoids the cascaded errors common in two-stage systems.
The PromptSpeech Dataset
To benchmark this task, the researchers created PromptSpeech, the first TTS dataset featuring natural language prompts.
The dataset includes a Synthesized Version with approximately 155,000 training samples generated via commercial APIs across five style factors. Experts wrote initial prompts, which were expanded using SimBERT to create diverse linguistic variations. A Real Version was also derived from LibriTTS, containing 26,000 samples covering four style factors (excluding emotion).
Key Results and Insights
Experiments demonstrated that PromptTTS is highly effective at both style control and maintaining audio fidelity:
- Style Accuracy: The system achieved an average style control accuracy of 90.31% on the synthesized dataset, outperforming two-stage baselines.
- Speech Quality: In Mean Opinion Score (MOS) tests, PromptTTS scored 4.00/5.00 on synthesized data and 3.80/5.00 on real data, indicating natural-sounding output.
- Semantic Extraction: Ablation studies confirmed that raw BERT models are insufficient for style extraction; fine-tuning (specifically P-tuning v2) is critical for the model to associate keywords like "cheerful" with specific acoustic properties.
Conclusion and Future Work
PromptTTS represents a significant shift from parameter tuning to prompt engineering in speech synthesis. While the current model is limited to the style categories present in the training data, future research will focus on zero-shot capabilities—allowing the model to interpret and synthesize styles it has never explicitly encountered. Additionally, expanding the diversity of real-world emotional datasets will further enhance the system's expressiveness.
Summary updated