Avaliação de arquiteturas de síntese de fala generativa com abordagens de espectrograma e fim-a-fim em cenários low-resource para clonagem de voz

Bruno Ribeiro, Gustavo H. dos S. Figueiredo, Leonardo H. da S. Correia, Mário Uliani Neto, Fernando O. Runstein, Ricardo P. V. Violato2024 · DOI

Abstract

O artigo compara modelos de síntese de fala com arquiteturas baseadas em espectrograma e fim-a-fim, com o objetivo de determinar a capacidade de clonagem de voz em cenário low-resource. Foram avaliados conjuntos de treinamento de adaptação com diferentes quantidades de fala para clonagem de uma voz alvo, e o tempo necessário para realizar o treinamento. O modelo VITS mostrou-se mais eficiente, alcançando os melhores resultados no teste de qualidade perceptual no cenário low-resource com dados no idioma português, e completou o treinamento em menos tempo, quando comparado com o Tacotron2.

Introduction

Creating high-quality Text-to-Speech (TTS) systems traditionally requires extensive datasets, often exceeding dozens of hours of professional recordings. However, in many practical applications—such as personalized digital assistants or voice restoration—only limited data (low-resource) is available. This study investigates the efficiency of two prominent generative architectures, Tacotron2 and VITS, in cloning a target voice for Brazilian Portuguese using as little as five minutes of audio.

Methodology

The research evaluates two distinct paradigms in speech synthesis:

  1. Spectrogram-based (Tacotron2): A two-stage pipeline where text is first converted into a mel-spectrogram, which is then transformed into a waveform by a separate vocoder (HiFi-GAN).
  2. End-to-End (VITS): A single-stage architecture using Variational Inference to map text directly to audio waveforms, bypassing intermediate spectrogram generation.

Training Strategy

The models underwent a two-phase training process:

  • Base Training: Models were pre-trained on the TTS-Portuguese Corpus (10.5 hours of a single male speaker) to learn linguistic fundamentals.
  • Fine-tuning (Cloning): The base models were adapted to a new target male speaker using subsets of a 20-minute proprietary dataset (CPQD), specifically testing 20, 15, 10, and 5-minute segments.

Results and Discussion

The evaluation combined objective metrics like Mel-Cepstral Distortion (MCD) with subjective human assessments. VITS consistently outperformed Tacotron2 across all data sizes.

Distribution of subjective rating scores across different stimuli and trial configurations.

As shown in the figure above, the subjective ratings (on a 0-100 scale) indicate that VITS (represented by the C2 stimulus) achieved higher median scores and lower variance compared to Tacotron2 (C1). While Tacotron2 often exhibited "glitches" or artifacts when trained on minimal data, VITS maintained a robust output quality. Specifically, with only 5 minutes of data, VITS achieved a score of 71.49, significantly higher than Tacotron2's 67.49.

Furthermore, VITS demonstrated superior efficiency in training time. High-quality cloning was achievable after just one hour of fine-tuning, peaking at 20 hours. This efficiency, combined with the lack of error accumulation inherent in two-stage systems, makes VITS a more practical solution for low-resource scenarios.

Conclusion

The study concludes that end-to-end architectures like VITS are significantly more effective for low-resource voice cloning in Brazilian Portuguese than traditional spectrogram-based models. VITS provides higher naturalness, better voice similarity, and faster convergence. Future research will explore cross-gender cloning and the impact of multi-speaker base models to further reduce the data requirements for high-fidelity voice synthesis.

Summary updated

Abstract

O artigo compara modelos de síntese de fala com arquiteturas baseadas em espectrograma e fim-a-fim, com o objetivo de determinar a capacidade de clonagem de voz em cenário low-resource. Foram avaliados conjuntos de treinamento de adaptação com diferentes quantidades de fala para clonagem de uma voz alvo, e o tempo necessário para realizar o treinamento. O modelo VITS mostrou-se mais eficiente, alcançando os melhores resultados no teste de qualidade perceptual no cenário low-resource com dados no idioma português, e completou o treinamento em menos tempo, quando comparado com o Tacotron2.
BibTeX
@misc{ribeiro2024avaliacao,
  author = {Bruno Ribeiro and Gustavo H. dos S. Figueiredo and Leonardo H. da S. Correia and Mário Uliani Neto and Fernando O. Runstein and Ricardo P. V. Violato and Mário Uliani Neto},
  title = {{Avaliação de arquiteturas de síntese de fala generativa com abordagens de espectrograma e fim-a-fim em cenários low-resource para clonagem de voz}},
  year = {2024},
  doi = {10.5753/stil.2024.245291},
}
Figure