ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation

Ruibo Fu, Xin Qi, Zhengqi Wen, Jianhua Tao, Tao Wang, Chunyu Qiang, Zhiyong Wang, Yi Lu, Xiaopeng Wang, Shuchen Shi, Yukun Liu, Xuefei Liu, Shuai ZhangarXiv 2024 · DOI · arXiv

Introduction

Speaker adaptation is a critical task in Text-to-Speech (TTS) synthesis, enabling models to mimic a target speaker's voice using limited reference audio. Existing techniques primarily rely on zero-shot encoding or model fine-tuning. Zero-shot methods, while fast, often fail to capture the nuanced timbre of the target speaker. Conversely, fine-tuning on sparse data (e.g., 1–5 sentences) frequently results in catastrophic overfitting, where the model gains speaker similarity at the expense of speech quality and intelligibility.

To address these limitations, we propose ASRRL-TTS, an Agile Speaker Representation Reinforcement Learning framework. Instead of modifying the underlying TTS model weights, ASRRL-TTS employs a Reinforcement Learning (RL) agent to search for an optimal speaker embedding that maximizes similarity, quality, and intelligibility.

Methodology

Framework Overview

The ASRRL-TTS framework decouples the speaker's timbre from the linguistic content by refining the speaker representation in a frozen TTS model. The process involves state initialization, an RL-based action design, and a multi-dimensional reward mechanism.

Reinforcement Learning for Embedding Optimization

The core of the system is an RL agent trained via Proximal Policy Optimization (PPO). The agent's state $s$ is defined by the concatenation of textual features $f_t$ (extracted via a pre-trained BERT model) and the initial speaker embedding $e$. This inclusion of textual context allows the agent to make context-aware adjustments to the speaker representation.

The agent performs actions based on the available data:

  • Single-Sentence (SS) Scenario: The agent generates a refinement vector added to the initial embedding.
  • Few-Sentence (FS) Scenario: When multiple reference clips are available, the agent learns dynamic fusion ratios to combine embeddings from multiple sentences, capturing a broader phonetic range.

Multi-Dimensional Fusion Reward

To ensure high-fidelity synthesis, the agent is guided by a weighted fusion reward $R$ comprising:

  1. Speaker Similarity: Evaluated using a pre-trained voiceprint model.
  2. Speech Quality: Predicted via a Mean Opinion Score (MOS) model.
  3. Intelligibility: Measured by Word Error Rate (WER) from an automatic speech recognition (ASR) system.

Experiments and Results

ASRRL-TTS was evaluated on LibriTTS and VCTK datasets using VITS and Grad-TTS architectures. Experimental results demonstrate significant improvements over traditional methods. In the single-sentence scenario, similarity scores increased by up to 30%, while few-sentence scenarios saw gains of up to 50%. Subjective evaluations showed MOS improvements of up to 0.25.

A key finding is the "20-sentence threshold": ASRRL-TTS significantly outperforms fine-tuning when reference data is below 20 sentences. Beyond this point, traditional fine-tuning becomes more effective as the risk of overfitting diminishes with more data.

Conclusion

ASRRL-TTS provides an agile and robust solution for low-resource speaker adaptation. By shifting the focus from weight fine-tuning to embedding space search, the framework achieves superior speaker similarity without sacrificing audio quality or intelligibility. Future work will explore extending this RL approach to adapt prosody and emotional expression.

Summary updated

Abstract

Speaker adaptation, which involves cloning voices from unseen speakers in the Text-to-Speech task, has garnered significant interest due to its numerous applications in multi-media fields. Despite recent advancements, existing methods often struggle with inadequate speaker representation accuracy and overfitting, particularly in limited reference speeches scenarios. To address these challenges, we propose an Agile Speaker Representation Reinforcement Learning strategy to enhance speaker similarity in speaker adaptation tasks. ASRRL is the first work to apply reinforcement learning to improve the modeling accuracy of speaker embeddings in speaker adaptation, addressing the challenge of decoupling voice content and timbre. Our approach introduces two action strategies tailored to different reference speeches scenarios. In the single-sentence scenario, a knowledge-oriented optimal routine searching RL method is employed to expedite the exploration and retrieval of refinement information on the fringe of speaker representations. In the few-sentence scenario, we utilize a dynamic RL method to adaptively fuse reference speeches, enhancing the robustness and accuracy of speaker modeling. To achieve optimal results in the target domain, a multi-scale fusion scoring mechanism based reward model that evaluates speaker similarity, speech quality, and intelligibility across three dimensions is proposed, ensuring that improvements in speaker similarity do not compromise speech quality or intelligibility. The experimental results on the LibriTTS and VCTK datasets within mainstream TTS frameworks demonstrate the extensibility and generalization capabilities of the proposed ASRRL method. The results indicate that the ASRRL method significantly outperforms traditional fine-tuning approaches, achieving higher speaker similarity and better overall speech quality with limited reference speeches.
BibTeX
@misc{fu2024asrrl,
  author = {Ruibo Fu and Xin Qi and Zhengqi Wen and Jianhua Tao and Tao Wang and Chunyu Qiang and Zhiyong Wang and Yi Lu and Xiaopeng Wang and Shuchen Shi and Yukun Liu and Xuefei Liu and Shuai Zhang},
  title = {{ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation}},
  year = {2024},
  eprint = {2407.05421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.05421},
}
Figure