OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech

Yong Ren, Jiangyan Yi, Jianhua Tao, Haiyang Sun, Zhengqi Wen, Hao Gu, Le Xu, Ye BaiarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces OV-InstructTTS, a novel paradigm for text-to-speech (TTS) that allows for more flexible and user-friendly control over speech synthesis using open-vocabulary natural language instructions. They achieve this by introducing a new dataset, OV-Speech, and a reasoning-driven framework that explicitly models the connection between high-level instructions and low-level acoustic features, leading to significantly improved instruction-following and expressiveness.

What Problem Does This Paper Address?

Current "InstructTTS" systems, which aim to control speech synthesis using natural language prompts, are limited. They typically rely on rephrased audio labels or predefined attributes, making it difficult to handle complex, high-level instructions that content creators might want to use. This rigid control hinders the ability to convey nuanced expressive intent. The paper addresses this gap by proposing a system that can interpret truly open-ended, descriptive instructions and translate them into expressive speech, moving beyond simple attribute manipulation.

Methods

The core of the proposed solution is the OV-InstructTTS-TEP framework. This framework operates in two main steps:

  1. Thinking Token Generation: Instead of directly mapping instructions to speech, the model first "thinks." It uses a Large Audio Language Model (LALM) backbone (specifically, Step-Audio-2-mini-Base) to generate a textual "reasoning chain." This chain deconstructs the high-level instruction, identifies implied contextual elements (like environment, speaker's personality, or intent), and infers specific emotional, acoustic, and paralinguistic attributes.
  2. Interleaved Text-Audio Token Generation: The model then synthesizes speech by generating an interleaved sequence of text and audio tokens. The text tokens are an enhanced version of the original transcript, augmented with the inferred emotional and paralinguistic tags. This structured output guides the subsequent audio generation, which uses a Flow-matching model and HiFiGAN vocoder.

This "reasoning-driven" approach explicitly bridges the semantic gap between abstract instructions and concrete acoustic realizations.

Datasets

The paper introduces a new dataset called OV-Speech. This dataset is built upon the ContextSpeech corpus (476.8 hours of multi-speaker audiobooks) and is augmented with:

  • Open-Vocabulary Instructions: Derived from narrative context, not just rephrased acoustic labels.
  • Reasoning Chains: Generated by LLMs, connecting high-level instructions to acoustic features.
  • Paralinguistic-Aware Transcriptions: Transcripts enriched with tags for non-speech vocalizations (e.g., [Laughter], [Cough]).

The construction of OV-Speech involves a five-stage pipeline: contextual information extraction, open-vocabulary instruction generation, consistency filtering, reasoning process annotation, and paralinguistic-aware transcription annotation.

Key Insights and Ideas

  • The Power of Explicit Reasoning: The paper strongly suggests that explicitly modeling a "thinking" or reasoning process is crucial for bridging the gap between abstract, open-vocabulary instructions and concrete speech synthesis. Simply trying to learn this mapping implicitly is insufficient.
  • Dataset Design Matters: The OV-Speech dataset, with its focus on narrative-derived instructions and explicit reasoning chains, is a key enabler for the proposed paradigm. It moves beyond the limitations of previous datasets that relied on simpler attribute labels.
  • LLMs as Reasoning Engines for TTS: The work highlights the potential of Large Language Models (and specifically Large Audio Language Models) not just for generating text or audio, but for performing complex reasoning tasks that are essential for advanced controllable TTS.
  • Synergy of Reasoning and Enriched Transcripts: While enriched transcripts (with emotion/paralinguistic tags) are helpful, their effectiveness is significantly amplified when combined with the explicit reasoning process. Reasoning helps the model correctly infer and utilize these fine-grained attributes.

Key Results

  • Superior Instruction Following: OV-InstructTTS-TEP achieved the highest Gemini Score (70.42) and best Gemini Rank (3.39/6) among evaluated systems, indicating significantly better adherence to open-vocabulary instructions.
  • Improved Naturalness and Expressiveness: Subjective evaluations showed the model achieved the best MOS (4.28) and ICMOS (3.91), outperforming baselines in both naturalness and instruction consistency.
  • Competitive Speaker Similarity: The model maintained a strong speaker similarity (SIM 0.722) with a competitive Character Error Rate (CER 3.61%).
  • Ablation Study Highlights Key Components: The ablation study confirmed that both the OV-Speech dataset and the reasoning component (T) were critical for performance gains. The combination of reasoning and enriched transcriptions (TEP) yielded the best results.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations, but the reliance on LLMs for reasoning and annotation implies potential biases or inaccuracies inherent in those models. The complexity of the reasoning process might also introduce computational overhead.
  • Future Work: The authors believe their work paves the way for more intuitive, flexible, and user-centric controllable TTS systems. Future directions likely involve further refining the reasoning process, exploring even more complex instruction types, and potentially reducing the computational cost of the framework. The introduction of the OV-Speech dataset also opens doors for further research in open-vocabulary InstructTTS.

Summary updated

Abstract

Instruct Text-to-Speech (InstructTTS) leverages natural language descriptions as style prompts to guide speech synthesis. However, existing InstructTTS methods mainly rely on a direct combination of audio-related labels or their diverse rephrasings, making it difficult to handle flexible, high-level instructions. Such rigid control is insufficient for users such as content creators who wish to steer generation with descriptive instructions. To address these constraints, we introduce OV-InstructTTS, a new paradigm for open-vocabulary InstructTTS. We propose a comprehensive solution comprising a newly curated dataset, OV-Speech, and a novel reasoning-driven framework. The OV-Speech dataset pairs speech with open-vocabulary instructions, each augmented with a reasoning process that connects high-level instructions to acoustic features. The reasoning-driven framework infers emotional, acoustic, and paralinguistic information from open-vocabulary instructions before synthesizing speech. Evaluations show that this reasoning-driven approach significantly improves instruction-following fidelity and speech expressiveness. We believe this work can inspire the next user-friendly InstructTTS systems with stronger generalization and real-world applicability. The dataset and demos are publicly available on our project page.
BibTeX
@misc{ren2026ov,
  author = {Yong Ren and Jiangyan Yi and Jianhua Tao and Haiyang Sun and Zhengqi Wen and Hao Gu and Le Xu and Ye Bai},
  title = {{OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech}},
  year = {2026},
  eprint = {2601.01459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.01459},
}
Figure