Noise Robust Cross-Speaker Emotion Transfer in TTS Through Knowledge Distillation and Orthogonal Constraint

Rui Liu, Kailin Liang, De Hu, Tao Li, Dongchao Yang, Haizhou LiIEEE Transactions on Audio, Speech and Language Processing 2025 · OpenReview

Abstract

By cross-speaker emotion transfer (CSEF) in text-to-speech (TTS) synthesis, we synthesize speech for a target speaker with the emotion transferred from reference speech by another (source) speaker. Traditionally CSEF is achieved by decoupling speaker and emotion components in speech, that highly depends on the availability of clean reference speech. To address the above issue, we propose a novel Noise-robust Cross-Speaker Emotion Transfer TTS model, i.e. NCE-TTS. NCE-TTS integrates the noise-robust emotion information extraction and noise-robust speaker-emotion disentanglement into a unified framework with two new modules, namely knowledge distillation and orthogonal constraint. The knowledge distillation aims to directly learn the emotion features of clean speech, from noisy speech, with a conditional diffusion model. The orthogonal constraint seeks to disentangle the deep emotion embedding and speaker embedding and further enhance the emotion-discriminative ability. Unlike the traditional cascaded approach of first denoising and then extracting features, we have built a new training framework that achieves better emotion transfer results in noisy scenarios. We conducted extensive experiments on a multi-speaker English emotional speech dataset ESD. The objective and subjective results demonstrate that the proposed NCE-TTS can synthesize emotionally rich speech while preserving the target speaker's voice in various noisy scenarios, with a significant improvement compared to all advanced baselines.
BibTeX
@article{liu2025noise,
  author = {Rui Liu and Kailin Liang and De Hu and Tao Li and Dongchao Yang and Haizhou Li},
  title = {{Noise Robust Cross-Speaker Emotion Transfer in TTS Through Knowledge Distillation and Orthogonal Constraint}},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=NqMNVCzrPN},
}

Loading related content...

Figure