Abstract
This paper focuses on generating speech with the acoustic style that meets users’ needs based on their open-domain instructions. To control the style, early work mostly relies on pre-defined rules or templates. The control types and formats are fixed in a closed domain, making it hard to meet diverse needs of users. One solution is to resort to instructions in free text to guide the generation. Current work mainly studies the instructions that clearly specify the acoustic styles, such as low pitch and fast speed. However, the instructions are complex, some even vague and abstract, such as “Generate a voice of a woman who is heartbroken due to a breakup.” It is hard to infer this implicit style by traditional matching-based methods. To address this problem, we propose a new controllable model. It first utilizes multimodal LLMs with knowledge-augmented techniques to infer the desired speech style from the instructions. The powerful language understanding ability of LLMs can help us better elicit the implicit style factors from the instruction. By using these factors as a control condition, we design a diffusion-based generator adept at finely adjusting speech details. That offers higher flexibility to meet complex users’ needs. Next, we verify the output speech from three aspects, i.e., consistency of decoding state, mel-spectrogram, and instruction style. This verified feedback can inversely optimize the generator. Extensive experiments are conducted on three popular datasets. The results show the effectiveness and good controllability of our approach.
Introduction
Traditional Text-to-Speech (TTS) systems typically operate in a "closed-domain" environment, relying on predefined templates or explicit parameters (e.g., specific pitch or speed values) to adjust acoustic styles. While effective for simple adjustments, these systems fail to bridge the "semantic-to-acoustic" divide when faced with natural human language. Users often provide instructions that are vague, emotional, or scenario-based, such as describing a voice as "gentle" or "heartbroken."
Mapping these abstract descriptions to specific acoustic features like pitch, energy, and duration is a non-linear and subjective challenge. This research addresses the gap by proposing a framework capable of interpreting open-domain instructions to facilitate fine-grained, controllable speech generation.
Methodology
The proposed architecture follows a three-step process designed to translate abstract intent into high-fidelity audio.
1. Instruction Encoding and Analysis
Instead of simple keyword matching, the system employs a Multimodal Large Language Model (MLLM). To mitigate ambiguity and "hallucinations" regarding acoustic styles, the model uses Retrieval-Augmented Generation (RAG). By retrieving the top-K most relevant audio-text pairs from a style reference dataset, the MLLM is grounded in real-world acoustic context, allowing it to output precise "style factors."
2. Controllable Speech Generation
These style factors, along with the speaking content and optional speaker voiceprint, are processed by a Variance Adaptor. This adaptor feeds into a Conditional Latent Diffusion Model. Diffusion models excel at learning complex mappings by iteratively refining noise into a signal, allowing for the fine adjustment of speech details that match the complex user instructions.
3. Multi-dimensional Verification
To ensure the output aligns with the user's prompt, an assessment module evaluates the generated speech across three dimensions:
- Decoding State Consistency: Stability of the internal VAE decoder representations.
- Mel-spectrogram Consistency: Structural similarity (SSIM) to ground truth features.
- Instruction Style Consistency: An MLLM-based check to verify if the generated audio matches the original prompt's "vibe." This verification provides a feedback loop that inversely optimizes the generator through fine-tuning.
Experiments and Results
The model was evaluated against six state-of-the-art baselines (including VoxInstruct and CosyVoice) across three datasets: PromptSpeech, SpeechCraft, and a newly introduced High-level Prompt Speech Corpus (HPSC) designed for complex, scenario-based instructions.
Quantitative results demonstrate that the proposed model consistently outperforms existing methods. On the HPSC dataset, the model achieved a 13.54% improvement in average quality metrics over the best baseline. Furthermore, style accuracy—measuring the model's ability to match age, gender, and pitch—improved by 3.73% to 9.21% across datasets. Qualitative human evaluations (QMOS, IMOS, RMOS) confirmed that the generated speech was perceived as more natural and more consistent with the provided instructions.
Conclusion
By combining the semantic power of MLLMs with the generative precision of diffusion models and a robust verification loop, this framework significantly advances the state of controllable TTS. While currently focused on English, future work aims to address the complexities of tonal languages and multi-lingual prosody to further broaden the system's applicability.
Summary updated