TDASS: Target Domain Adaptation Speech Synthesis Framework for Multi-speaker Low-Resource TTS

Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

Introduction

Modern Text-to-Speech (TTS) systems, such as Tacotron2 and FastSpeech, have achieved remarkable naturalness by training on massive datasets. However, these models are typically data-hungry, requiring hundreds or thousands of clean utterances per speaker to achieve high fidelity. In many practical applications, researchers only have access to "low-resource" datasets, often consisting of fewer than 100 sentences for a specific target speaker.

When multi-speaker models are used to bridge this data gap, they often suffer from a "voice similarity" problem: the synthesized voice of the target speaker often sounds like a generic blend of the pre-training speakers or loses the unique timbre of the target. This paper introduces TDASS (Target Domain Adaptation Speech Synthesis), a framework designed to maintain high voice similarity and naturalness even when the target speaker's data is extremely limited.

Methodology

TDASS is built upon the Tacotron2 architecture, specifically enhancing the "Generate Network" to better isolate and reinforce target speaker characteristics. The core innovation consists of two integrated components: the Self-interested Classifier and a specialized Gradient Reversal Layer (GRL).

The Self-interested Classifier

This module operates on the hidden features of the network, which contain a mixture of linguistic information and speaker timbre. The classifier is trained to distinguish whether a given feature set belongs to the "target" speaker or a "non-target" speaker. By identifying these differences, the model can more effectively partition the latent space.

Specialized Gradient Reversal Layer (GRL)

The training process utilizes a modified GRL to treat speakers asymmetrically:

  1. Non-target Speakers: For these speakers, the GRL performs adversarial training. It reverses the gradients during backpropagation to "remove" or "unlearn" their specific stylistic influences from the shared encoding. This prevents the target speaker's voice from being contaminated by the traits of the high-resource speakers.
  2. Target Speaker: For the target speaker, the model uses standard supervised learning. This reinforces the specific nuances, style, and timbre of the target voice, ensuring the model "obsesses" over the limited data available.

By combining these approaches, TDASS learns to ignore the stylistic "noise" of the background speakers while maximizing the extraction of the target speaker's unique identity.

Experimental Setup

The researchers evaluated the framework using a Chinese speech corpus featuring three speakers:

  • Pre-training Set: Two non-target speakers providing a combined total of over 24,000 utterances.
  • Target Set: A third speaker used for low-resource fine-tuning.
  • Benchmarks: The model was tested across four data scales for the target speaker: 30, 100, 300, and 500 utterances. TDASS was compared against a standard Tacotron2 baseline.

Results and Discussion

The performance was measured using Mean Opinion Score (MOS) for naturalness, Voice Similarity Score (VSS) for speaker identity, and Mel Cepstral Distortion (MCD) for objective wave accuracy.

Superiority in Extreme Scarcity

TDASS demonstrated significant gains in the most data-constrained scenarios. With only 30 utterances, TDASS achieved a MOS of 3.51, compared to the Tacotron2 baseline of 3.06. Similarly, the VSS for TDASS was approximately 0.7 points higher than the baseline at this scale, indicating a much stronger resemblance to the original speaker.

Data Efficiency

One of the most striking findings was the efficiency of the framework. TDASS trained on just 100 utterances performed comparably to, or in some cases better than, the baseline Tacotron2 model trained on 500 utterances. This suggests that the adversarial filtering mechanism effectively compensates for a 5x lack of data.

Objective Metrics

Mel Cepstral Distortion (MCD) scores, which measure the spectral distance between synthesized and ground-truth audio, were consistently lower for TDASS across all test cases. This objective improvement confirms that the synthesized waveforms are structurally closer to the target speaker's actual speech patterns.

Conclusion and Future Work

TDASS provides a robust solution for personalized TTS in low-resource environments by explicitly filtering out the influence of non-target speakers. The "self-interested" approach to domain adaptation ensures that even with as few as 30 sentences, the model can capture a speaker's unique timbre without it being diluted by the broader training set.

Future research may explore how this framework scales when the non-target pool includes hundreds of diverse speakers, as well as its performance in cross-lingual scenarios. Additionally, integrating this logic into end-to-end architectures that bypass the need for separate vocoders could further streamline the synthesis process.

Summary updated

BibTeX
@inproceedings{zhang2022tdass,
  author = {Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{TDASS: Target Domain Adaptation Speech Synthesis Framework for Multi-speaker Low-Resource TTS}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=CQcpdnyYCY},
}
Figure