Abstract
Any-to-any voice conversion aims to convert the voice from and to any speakers even unseen during training, which is much more challenging compared to one-to-one or many-to-many tasks, but much more attractive in real-world scenarios. In this paper we proposed FragmentVC, in which the latent phonetic structure of the utterance from the source speaker is obtained from Wav2Vec 2.0, while the spectral features of the utterance(s) from the target speaker are obtained from log mel-spectrograms. By aligning the hidden structures of the two different feature spaces with a two-stage training process, FragmentVC is able to extract fine-grained voice fragments from the target speaker utterance(s) and fuse them into the desired utterance, all based on the attention mechanism of Transformer as verified with analysis on attention maps, and is accomplished end-to-end. This approach is trained with reconstruction loss only without any disentanglement considerations between content and speaker information and doesn't require parallel data. Objective evaluation based on speaker verification and subjective evaluation with MOS both showed that this approach outperformed SOTA approaches, such as AdaIN-VC and AutoVC.
Introduction
Voice Conversion (VC) aims to modify the speaker identity of an audio signal while preserving its linguistic content. The most challenging variant is any-to-any conversion, where both source and target speakers may be unseen during training. Traditional state-of-the-art models typically rely on disentangling "content" from "style" (speaker identity). However, perfect disentanglement is notoriously difficult to achieve, often leading to information leakage or degraded audio quality.
FragmentVC introduces a novel paradigm inspired by concatenative synthesis. Instead of explicit disentanglement, it uses a Transformer-based attention mechanism to "pick and choose" fine-grained spectral fragments from a target speaker's utterances that match the phonetic structure of the source speech.
Methodology
The FragmentVC architecture is composed of three primary modules:
- Source Encoder: Utilizes a frozen, pre-trained Wav2Vec 2.0 model to extract rich, latent phonetic representations from the source audio.
- Target Encoder: A series of 1D-convolutional layers that process log mel-spectrograms of the target speaker's reference utterances to extract spectral features.
- Decoder (Extraction and Fusion): The core component consisting of a stack of Extractors and Smoothers.
- Extractors use cross-attention where source phonetic features act as queries to search and retrieve matching spectral fragments (keys and values) from the target encoder.
- Smoothers employ self-attention to ensure natural transitions between the fused fragments.
The model is trained in two stages. Initially, it learns to reconstruct a single utterance. In the second stage, the target input is expanded to multiple utterances from the same speaker, forcing the model to learn robust extraction and fusion from a larger pool of voice fragments.
Attention as Unit Selection
A key insight of FragmentVC is that the cross-attention mechanism implicitly performs fine-grained unit selection. By analyzing the attention maps, the authors demonstrate that the model effectively aligns phonetic structures across different speakers.
To prevent the source speaker's identity from leaking into the output, the first Extractor layer omits the residual connection. This forces the decoder to rely entirely on the fragments retrieved from the target speaker.
Experimental Results
FragmentVC was evaluated on the CSTR VCTK Corpus and the CMU Arctic dataset. It demonstrated superior performance compared to SOTA baselines like AdaIN-VC and AUTOVC, particularly in "unseen-to-unseen" (u2u) scenarios.
- Speaker Verification (SV) Accuracy: In u2u tests, FragmentVC achieved 92.5% accuracy, significantly higher than AdaIN-VC (87.1%) and AUTOVC (19.0%).
- Subjective Quality: Mean Opinion Scores (MOS) for both naturalness and speaker similarity consistently favored FragmentVC.
- Data Efficiency: The model remains effective even with a single target utterance (86.5% SV accuracy), though performance scales with more reference data.
Conclusion
FragmentVC successfully bypasses the disentanglement bottleneck by framing voice conversion as an attention-based fragment extraction and fusion task. While the model achieves high similarity and quality, future work will address the trade-off between naturalness and similarity observed in multi-stage training and explore alternative phonetic representations to further enhance performance.
Summary updated