Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Data

Tan Lee 李丹, Yusheng Tian, Wei Liu, Tan LeeAsru 2023

TL;DR

This paper introduces DMSEtext, a novel diffusion-based model for enhancing low-quality speech recordings to improve personalized text-to-speech (TTS) synthesis. By performing enhancement directly on Mel-spectrograms and conditioning on text transcriptions, DMSEtext significantly boosts the quality of synthetic voices derived from limited, degraded "found" data.

What Problem Does This Paper Address?

Creating personalized synthetic voices for Text-to-Speech (TTS) systems typically relies on large, high-quality, carefully recorded datasets. However, in many real-world scenarios, only "found data" – recordings that were not specifically collected for TTS – are available. This found data is often plagued by various audio degradations like background noise, reverberation, clipping, and band limiting, making it unsuitable for direct TTS training. While previous methods have attempted to address this by either augmenting TTS models with noise embeddings or using separate enhancement models, they often struggle with multiple degradation types simultaneously or produce outputs not optimally aligned with TTS objectives. This paper tackles the challenge of effectively enhancing diverse, degraded found speech data to enable high-quality personalized TTS synthesis.

Methods

The core of this work is a conditional diffusion model for Mel-spectrogram enhancement. Diffusion models are generative models that learn to reverse a gradual noise-adding process. Here, the authors adapt this to enhance degraded Mel-spectrograms (y) to produce clean ones (x).

Key methodological aspects include:

  • Mel-Spectrogram Domain Enhancement: Instead of operating on raw audio waveforms, the enhancement is performed directly on log Mel-spectrograms. This is a crucial design choice as Mel-spectrograms are the standard input for many TTS acoustic models, ensuring better alignment with the downstream TTS training objective.
  • Conditional Diffusion: The diffusion model is conditioned on the degraded Mel-spectrogram (y) to guide the generation of the clean version.
  • Text Conditioning: To improve robustness against unseen degradation types and enhance generalization, the model is further conditioned on text transcriptions. This is achieved by generating an average Mel-spectrogram representation (μ) from the text, which is then incorporated into the diffusion model's conditioning mechanism. This leverages the fact that text transcriptions are usually available during TTS development.
  • Variance Preserving (VP) Diffusion: The paper utilizes the Variance Preserving (VP) diffusion model framework, which defines a forward diffusion process (adding noise) and a reverse process (denoising to generate data).
  • Score Estimation: The diffusion model learns to estimate the "score" of the data distribution, which is essential for simulating the reverse denoising process. A neural network (Sθ) is trained to predict this score.

Datasets

  • Found Data: The primary dataset for evaluating the TTS system consists of 513 Cantonese utterances (approx. 37 minutes) from a target speaker who lost their voice. This data exhibits various degradations including background noise, reverberation, band limiting, and magnitude clipping.
  • Training Data for Enhancement Models:
    • DenoiseCUSENT: This dataset was created by applying Demucs (a strong denoising model) to the CUSENT dataset (a multi-speaker Cantonese speech corpus). This provides high-quality clean references for training the enhancement models.
  • Simulated Degraded Data: To train the enhancement models to handle specific degradations, the authors synthesized distorted audio by applying noise (DNS dataset), reverberation (RIRs dataset), clipping, and band limiting to clean speech from CUSENT.
  • VCTK Dataset: A subset of the VCTK corpus (English multi-speaker corpus) was used to train a baseline diffusion model (DMSEbase(VCTK)) to rule out the impact of language mismatch when comparing with English-trained enhancement models.
  • Training Data for TTS Model: The TTS model was pre-trained on CUSENT and then fine-tuned on the enhanced found data of the target speaker.
  • ASR Evaluation Data: A commercial Cantonese speech recognition corpus (KingASR086) was used to train an ASR model for objective evaluation of speech enhancement.

Key Insights and Ideas

  • Mel-Spectrogram Enhancement is Key for TTS: Performing speech enhancement directly in the Mel-spectrogram domain, rather than the waveform, is a critical design choice that significantly benefits downstream TTS synthesis. This aligns the enhancement output with the input requirements of TTS acoustic models.
  • Text Conditioning Boosts Robustness: Adding text transcriptions as a condition to the diffusion model dramatically improves its ability to generalize to unseen audio degradations. This is a significant finding for creating more robust enhancement systems.
  • Generative Models Outperform Regression for Cleanliness: Diffusion-based generative models, compared to regression-based methods like VoiceFixer, tend to produce cleaner speech outputs. This is attributed to the generative modeling approach, which is influenced by a clean speech prior.
  • Generalist Enhancement is Superior for Found Data: A single, generalist enhancement model capable of handling multiple degradation types (like the proposed DMSEtext) is more effective for found data TTS than single-task denoising models (like Demucs) or regression-based approaches.

Key Results

  • Objective Evaluation (ASR-based): The proposed DMSEtext model achieved the lowest Phone Error Rate (PER%) of 17.6% on the ASR evaluation, significantly outperforming baselines like Demucs (20.3%), VoiceFixer (29.7%), and DMSEbase (24.7%). This indicates better preservation of speech content and fewer distortions.
  • Subjective Evaluation (MOS):
    • Enhanced Speech Cleanliness: DMSEtext achieved the highest MOS score for cleanliness (4.12 ± 0.09), indicating superior removal of noise and reverberation.
  • Overall Impression: DMSEtext also received the highest MOS score for overall impression (4.02 ± 0.09) when used to synthesize speech for TTS. This signifies that the resulting synthetic voices were perceived as the most natural and pleasing.
  • TTS Performance: Synthetic voices trained on DMSEtext-enhanced data received significantly higher MOS scores for both cleanliness and overall impression compared to voices trained on data enhanced by Demucs, VoiceFixer, or DMSEbase.

Limitations and Future Work

  • Domain Gap: The authors acknowledge that a domain gap can still exist between the synthetic training data and real-world degraded recordings, potentially leading to overfitting. Text conditioning helps mitigate this, but it's an ongoing challenge.
  • Computational Cost: Diffusion models are known for their computational intensity, particularly during inference. While the paper mentions using an ODE solver (DPM-solver) for faster sampling, this remains a consideration.
  • Language Specificity: While DMSEbase(VCTK) showed comparable performance to DMSEbase on English data, the primary success of DMSEtext is demonstrated on Cantonese. Further exploration into cross-lingual generalization for text-conditioned enhancement could be a future direction.
  • Found Data Diversity: The study focuses on a specific speaker's found data. Evaluating the model's performance on a wider variety of speakers and degradation types would be beneficial.

Summary updated

Abstract

Creating synthetic voices with found data is challenging, as real-world recordings often contain various types of audio degradation. One way to address this problem is to pre-enhance the speech with an enhancement model and then use the enhanced data for text-to-speech (TTS) model training. This paper investigates the use of conditional diffusion models for generalized speech enhancement, which aims at addressing multiple types of audio degradation simultaneously. The enhancement is performed on the log Mel-spectrogram domain to align with the TTS training objective. Text information is introduced as an additional condition to improve the model robustness. Experiments on real-world recordings demonstrate that the synthetic voice built on data enhanced by the proposed model produces higher-quality synthetic speech, compared to those trained on data enhanced by strong baselines. Code and pre-trained parameters of the proposed enhancement model are available at \url{https://github.com/dmse4tts/DMSE4TTS}
BibTeX
@inproceedings{2023diffusion,
  author = {Tan Lee 李丹 and Yusheng Tian and Wei Liu and Tan Lee},
  title = {{Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Data}},
  booktitle = {Asru},
  year = {2023},
}
Figure