Introduction
Prosody control remains a significant challenge in Text-to-Speech (TTS) synthesis. While modern models excel at replicating a specific speaker's vocal identity, capturing the nuanced "delivery style"—including tempo, intensity, and rhythm—often requires complex architectures or vast datasets. This paper explores a novel, simplified approach to speech feature transfer by manipulating the attention mechanism within a Tacotron-based framework.
Methodology
The proposed method leverages the Tacotron architecture, specifically focusing on the attention module that aligns text embeddings with acoustic features. The core hypothesis is that attention weights encode not just alignment, but also the behavioral characteristics of a speaker's performance.
The transfer process involves five distinct stages:
- Model Training: Independent Tacotron models are trained for various speakers to capture their unique text-to-audio mapping.
- Attention Extraction: During inference, attention weights are extracted from a source speaker's model rather than the target's.
- Weight Injection: These source weights are injected into the target speaker's synthesis pipeline.
- Feature Transfer: The target model's decoder is forced to follow the timing and emphasis patterns of the source speaker.
- Synthesis: The final audio retains the target's vocal timbre but adopts the source's prosodic "vibe."
Experimental Setup
The study utilized three datasets to ensure a variety of speech styles:
- Benedict Cumberbatch Audiobooks: ~13 hours of professional narration.
- LJ Speech: A standard single-speaker public domain dataset.
- World English Bible (WEB): A diverse religious text dataset.
After preprocessing, the Cumberbatch model was trained on approximately 11 hours of high-quality audio segments.
Results and Discussion
The efficacy of the attention-swapping technique was evaluated through both visual waveform analysis and quantitative similarity metrics.
Visual Analysis
Waveform comparisons demonstrate a clear shift in the target speaker's output toward the source speaker's characteristics. When transferring features related to speech strength, the amplitude peaks of the target speaker align closely with those of the source.
Similarly, the transfer of speech speed is evident in the temporal alignment of the synthesized waveforms. By injecting the source's attention weights, the target speaker mimics the source's specific pacing and pauses.
Quantitative Evaluation
The proposed method showed significant improvements across five similarity metrics compared to baseline models:
- Euclidean Distance: 45.55% improvement.
- Manhattan Distance: 55.71% improvement.
- Cosine Similarity: Increased by 0.3984.
- Pearson Correlation: Increased by 0.2957.
These results confirm that the attention mechanism serves as an effective proxy for style, allowing for the decoupling of vocal identity from delivery style.
Conclusion and Future Work
This research presents a simple yet effective method for prosody transfer in TTS. By treating attention as a style proxy, the authors demonstrate that complex style encoders may not always be necessary for basic feature transfer. Future research will focus on expanding this technique to universal multi-speaker models and incorporating additional prosodic features such as pitch and emotional nuances.
Summary updated