Attention-based speech feature transfer between speakers

Hangbok Lee, Minjae Cho, Hyuk-Yoon KwonFrontiers Artif. Intell 2024 · DOI · OpenReview

Introduction

Prosody control remains a significant challenge in Text-to-Speech (TTS) synthesis. While modern models excel at replicating a specific speaker's vocal identity, capturing the nuanced "delivery style"—including tempo, intensity, and rhythm—often requires complex architectures or vast datasets. This paper explores a novel, simplified approach to speech feature transfer by manipulating the attention mechanism within a Tacotron-based framework.

Methodology

The proposed method leverages the Tacotron architecture, specifically focusing on the attention module that aligns text embeddings with acoustic features. The core hypothesis is that attention weights encode not just alignment, but also the behavioral characteristics of a speaker's performance.

The transfer process involves five distinct stages:

  1. Model Training: Independent Tacotron models are trained for various speakers to capture their unique text-to-audio mapping.
  2. Attention Extraction: During inference, attention weights are extracted from a source speaker's model rather than the target's.
  3. Weight Injection: These source weights are injected into the target speaker's synthesis pipeline.
  4. Feature Transfer: The target model's decoder is forced to follow the timing and emphasis patterns of the source speaker.
  5. Synthesis: The final audio retains the target's vocal timbre but adopts the source's prosodic "vibe."

Experimental Setup

The study utilized three datasets to ensure a variety of speech styles:

  • Benedict Cumberbatch Audiobooks: ~13 hours of professional narration.
  • LJ Speech: A standard single-speaker public domain dataset.
  • World English Bible (WEB): A diverse religious text dataset.

After preprocessing, the Cumberbatch model was trained on approximately 11 hours of high-quality audio segments.

Results and Discussion

The efficacy of the attention-swapping technique was evaluated through both visual waveform analysis and quantitative similarity metrics.

Visual Analysis

Waveform comparisons demonstrate a clear shift in the target speaker's output toward the source speaker's characteristics. When transferring features related to speech strength, the amplitude peaks of the target speaker align closely with those of the source.

Waveform analysis showing the transfer of speech strength (amplitude) from a source speaker to a target speaker.

Similarly, the transfer of speech speed is evident in the temporal alignment of the synthesized waveforms. By injecting the source's attention weights, the target speaker mimics the source's specific pacing and pauses.

Waveform analysis demonstrating the effect of transferred attention weights on speech speed and timing.

Quantitative Evaluation

The proposed method showed significant improvements across five similarity metrics compared to baseline models:

  • Euclidean Distance: 45.55% improvement.
  • Manhattan Distance: 55.71% improvement.
  • Cosine Similarity: Increased by 0.3984.
  • Pearson Correlation: Increased by 0.2957.

These results confirm that the attention mechanism serves as an effective proxy for style, allowing for the decoupling of vocal identity from delivery style.

Conclusion and Future Work

This research presents a simple yet effective method for prosody transfer in TTS. By treating attention as a style proxy, the authors demonstrate that complex style encoders may not always be necessary for basic feature transfer. Future research will focus on expanding this technique to universal multi-speaker models and incorporating additional prosodic features such as pitch and emotional nuances.

Summary updated

Abstract

p>In this study, we propose a simple yet effective method for incorporating the source speaker's characteristics in the target speaker's speech. This allows our model to generate the speech of the target speaker with the style of the source speaker. To achieve this, we focus on the attention model within the speech synthesis model, which learns various speaker features such as spectrogram, pitch, intensity, formant, pulse, and voice breaks. The model is trained separately using datasets specific to the source and target speakers. Subsequently, we replace the attention weights learned from the source speaker's dataset with the attention weights from the target speaker's model. Finally, by providing new input texts to the target model, we generate the speech of the target speaker with the styles of the source speaker. We validate the effectiveness of our model through similarity analysis utilizing five evaluation metrics and showcase real-world examples.</p>
BibTeX
@article{lee2024attention,
  author = {Hangbok Lee and Minjae Cho and Hyuk-Yoon Kwon},
  title = {{Attention-based speech feature transfer between speakers}},
  journal = {Frontiers Artif. Intell.},
  year = {2024},
  doi = {10.3389/frai.2024.1259641},
}
Figure