Abstract
In recent years, speech generation has seen remarkable progress, now achieving one-shot generation capability that is often virtually indistinguishable from real human voice. Integrating such advancements in speech generation with large language models might revolutionize a wide range of applications. However, certain applications, such as assistive conversational systems, require natural and conversational speech generation tools that also operate efficiently in real time. Current state-of-the-art models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs, require large neural components and extensive training data to work well. In contrast, MQTTS aims to build more compact conversational TTS models while capitalizing on smaller-scale real-life conversational speech data. However, its autoregressive nature yields high inference latency and thus limits its real-time usage. In order to mitigate the current limitations of the state-of-the-art TTS models while capitalizing on their strengths, in this work we introduce the Pheme model series that 1) offers compact yet high-performing models, 2) allows for parallel speech generation of 3) natural conversational speech, and 4) it can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models. We also show that through simple teacher-student distillation we can meet significant improvements in voice quality for single-speaker setups on top of pretrained Pheme checkpoints, relying solely on synthetic speech generated by much larger teacher models. Audio samples and pretrained models are available online.
Introduction
While AI voices have become increasingly realistic, most high-end models like VALL-E or SoundStorm are resource-intensive, requiring tens of thousands of hours of training data and massive neural components. Furthermore, these models often struggle with the nuances of real-world conversation—including fillers, varying prosody, and background noise—compared to clean audiobook recordings.
Many existing conversational models are autoregressive, generating audio one token at a time. This creates high latency, making them difficult to use in real-time applications like voice assistants where delays are detrimental. To address these limitations, we introduce Pheme, a model series designed to be compact, data-efficient, and fast enough for true real-time interaction.
Methodology
Pheme follows a two-stage architecture for speech generation, introducing several key efficiencies:
- Unified Speech Tokenization: The model utilizes a SpeechTokenizer with Residual Vector Quantization (RVQ) to disentangle semantics (content) from acoustics (voice and emotion). The first layer of tokens handles the "what," while subsequent layers handle the "how."
- Text-to-Semantics (T2S): A compact T5-style encoder-decoder (50M–100M parameters) translates raw text into semantic tokens.
- Acoustics-to-Speech (A2S) with Parallel Decoding: Instead of sequential generation, Pheme uses a MaskGIT-style parallel decoding scheme. It iteratively fills in "masked" sequences based on confidence scores, allowing for high-quality audio generation in just a few iterations.
- Speaker Embeddings: To maintain voice consistency, the model incorporates speaker embeddings from a pre-trained diarization model (pyannote), facilitating effective one-shot voice cloning.
Experimental Setup
The models were trained primarily on a filtered 550-hour subset of the GigaSpeech dataset, focusing on high-quality conversational speech from podcasts and YouTube. Larger versions (300M parameters) also utilized LibriTTS and Multilingual LibriSpeech (MLS). Additionally, synthetic data generated by larger teacher models was used for single-speaker specialization through distillation, proving that teacher-student distillation works well for TTS.
Key Results
Pheme demonstrates significant improvements in both efficiency and speech quality compared to autoregressive baselines like MQTTS.
Speech Quality and Intelligibility
As shown in Table 1, Pheme-Small (100M) outperforms MQTTS in Word Error Rate (WER) and Mel-cepstral Distortion (MCD), indicating higher intelligibility and better acoustic accuracy despite being trained on significantly less data.
Inference Speed
The most notable achievement is the reduction in latency. On an NVIDIA A100 GPU, Pheme-Small achieved a Real-Time Factor (RTF) of 0.133, representing a roughly 14.5x speedup over the autoregressive MQTTS model (RTF 1.930).
Discussion and Conclusion
Our findings indicate that once audio generation (A2S) is parallelized, the text-to-semantic (T2S) stage becomes the primary bottleneck, accounting for approximately 90% of processing time. Furthermore, the results suggest that data quality is more critical than quantity; Pheme matches the quality of much larger models despite using 10x less training data. Future work will focus on reducing T2S latency through techniques like speculative decoding, expanding to multilingual support, and improving conversational data curation to capture more complex human interactions.
Summary updated