BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization

Henry B. Moss, Vatsal Aggarwal, Nishant Prateek, Javier González, Roberto Barra-Chicote

Abstract

We present BOFFIN TTS (Bayesian Optimization For FIne-tuning Neural Text To Speech), a novel approach for few-shot speaker adaptation. Here, the task is to fine-tune a pre-trained TTS model to mimic a new speaker using a small corpus of target utterances. We demonstrate that there does not exist a one-size-fits-all adaptation strategy, with convincing synthesis requiring a corpus-specific configuration of the hyper-parameters that control fine-tuning. By using Bayesian optimization to efficiently optimize these hyper-parameter values for a target speaker, we are able to perform adaptation with an average 30% improvement in speaker similarity over standard techniques. Results indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new speakers using less than ten minutes of audio, achieving the same naturalness as produced for the speakers used to train the base model.

TL;DR

This paper introduces BOFFIN TTS, a novel approach to few-shot speaker adaptation that uses Bayesian Optimization to automatically find the best fine-tuning hyper-parameters for individual target speakers. By proving that a "one-size-fits-all" adaptation strategy is fundamentally flawed, the authors achieve up to a 57% improvement in speaker similarity using less than 10 minutes of audio, without sacrificing the naturalness of the synthesized speech.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems can generate incredibly realistic speech, but training a new voice from scratch requires tens of hours of high-quality audio. To scale TTS to new voices with limited data (e.g., just a few minutes of audio), researchers use few-shot speaker adaptation—fine-tuning a pre-trained multi-speaker model on the new target voice.

However, few-shot adaptation is highly susceptible to overfitting and "catastrophic forgetting" (where the model learns the new voice but forgets how to pronounce words naturally). Traditionally, researchers use a fixed set of hyper-parameters (learning rate, dropout, etc.) for all new speakers. This paper tackles the reality that different target speakers—with varying acoustic properties, phonetic coverage, and audio quality—require entirely different regularization and learning dynamics to be cloned successfully.

Methods

The authors build on a standard Tacotron2-style sequence-to-sequence acoustic model paired with a neural vocoder. To adapt this base model to a new speaker, they freeze the encoder and attention modules (which handle general language and pronunciation) and only fine-tune the speaker embedding and decoder modules.

Architecture of the TTS model highlighting frozen weights during adaptation.

The core innovation is the BOFFIN TTS framework, which treats speaker adaptation as a hyper-parameter optimization (HPO) problem. Because evaluating a single hyper-parameter configuration is computationally expensive, the authors use Bayesian Optimization (BO) with a Gaussian Process model to intelligently select the most promising hyper-parameter configurations to test next.

BOFFIN TTS tunes 9 hyper-parameters per speaker. Seven are standard (learning rate, batch size, decay-factor, gradient-clipping, dropout, and two zoneout parameters). The authors also introduce two novel parameters specifically for adaptation:

  1. Mixing Ratio: A rehearsal method that controls the ratio of target speaker data to original base-model speaker data during fine-tuning.
  2. Starting Epoch: Determines which specific checkpoint of the pre-trained base model to start fine-tuning from.

Key Insights and Ideas

  • The Myth of "One-Size-Fits-All": The most striking finding is that optimal hyper-parameter configurations vary wildly not just across different datasets, but across individual speakers within the same dataset. A fixed adaptation strategy leaves significant performance on the table.
  • Bayesian Optimization Shines in TTS: This paper demonstrates that BO can navigate a complex, 9-dimensional search space to find highly performant, speaker-specific configurations in very few iterations, vastly outperforming Random Search.
Optimization curves and distribution of tuned hyper-parameters across different speakers and datasets.

Key Results

  • Overall Performance: BOFFIN TTS achieved an average 30% improvement in speaker similarity over standard baseline adaptation techniques, requiring less than 10 minutes of target audio (~100 utterances).
  • INTERNAL Dataset: Achieved a 28% improvement in speaker similarity over the baseline.
  • VCTK Dataset: On this more challenging, expressive dataset, BOFFIN TTS achieved a massive 57% improvement in similarity and a 13% improvement in naturalness over the baseline.
  • Maintained Quality: Across successful adaptations, BOFFIN TTS maintained Mean Opinion Score (MOS) naturalness ratings comparable to the original speakers used to train the base model.

Limitations and Future Work

Limitations: The framework's success is heavily bottlenecked by the quality of the base model. In the LibriTTS experiment, where the base model was trained on lower-quality audio-book data, human listeners could not discern a statistically significant preference in similarity between BOFFIN TTS and the baseline.

Future Work: The authors strongly advocate for the broader adoption of Bayesian Optimization across the TTS field to unlock significant performance gains in model fine-tuning.

Summary updated

Abstract

We present BOFFIN TTS (Bayesian Optimization For FIne-tuning Neural Text To Speech), a novel approach for few-shot speaker adaptation. Here, the task is to fine-tune a pre-trained TTS model to mimic a new speaker using a small corpus of target utterances. We demonstrate that there does not exist a one-size-fits-all adaptation strategy, with convincing synthesis requiring a corpus-specific configuration of the hyper-parameters that control fine-tuning. By using Bayesian optimization to efficiently optimize these hyper-parameter values for a target speaker, we are able to perform adaptation with an average 30% improvement in speaker similarity over standard techniques. Results indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new speakers using less than ten minutes of audio, achieving the same naturalness as produced for the speakers used to train the base model.
BibTeX
@inproceedings{moss2020boffin,
  author = {Henry B. Moss and Vatsal Aggarwal and Nishant Prateek and Javier González and Roberto Barra-Chicote},
  title = {{BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OFq-lDitcZr},
}
Figure