Introduction
Voice conversion (VC) aims to transform the speaker identity of a source utterance into that of a target speaker while preserving the original linguistic content. While modern "any-to-any" systems have made significant strides, they often face a critical trade-off: achieving high speaker similarity typically requires substantial target audio, whereas limited data (one-shot conversion) often leads to a collapse in intelligibility or poor voice mimicry.
The Phoneme Hallucinator addresses this by "hallucinating" a diverse set of synthetic phoneme representations for a target speaker. By expanding a mere 3-second sample into a rich library of sounds, the framework achieves state-of-the-art performance without requiring text annotations.
The Challenge of One-Shot Conversion
A prominent baseline in this field is kNN-VC, which employs a "search and replace" strategy. It maps source speech frames to the most similar frames in the target speaker's reference set. While effective with ample data (e.g., 10 minutes), its performance degrades sharply with short target samples. If the source speaker utters a phoneme not present in the brief target clip, the system fails to find a suitable match, resulting in a massive spike in Word Error Rate (WER) and garbled output.
The Phoneme Hallucinator Framework
The core innovation of this work is the transition from sequence generation to set expansion. Instead of trying to generate a coherent stream of audio, the model generates an unordered "bag of sounds" that represents the target speaker's phonetic range.
The pipeline consists of four main stages:
- Feature Extraction: Raw audio is processed via WavLM, a self-supervised model that captures high-dimensional representations of content and identity.
- The Hallucinator: A conditional generative model based on De Finetti’s theorem. It uses Set Transformers and a Conditional Variational Autoencoder (cVAE) to capture the global essence of a speaker's voice and sample an infinite variety of new frames.
- kNN Regression: The source speech features are matched against the expanded "hallucinated" set.
- Synthesis: A HiFi-GAN vocoder converts the resulting features back into audible speech.
Experimental Results
Evaluations on the LibriSpeech dataset demonstrate that the Phoneme Hallucinator maintains high performance even when target audio is extremely scarce.
As shown in Figure 1, while kNN-VC's error rate skyrockets as target duration decreases, the Phoneme Hallucinator remains stable. At a 3-second target duration, it achieves a 5.10% WER, compared to 45.92% for kNN-VC. Speaker similarity also remains significantly higher. Furthermore, the system is highly efficient; on an NVIDIA RTX 4090, it can hallucinate 10 minutes of speech representations in just 1.48 seconds.
Conclusion
By reframing voice conversion as a set expansion problem, the Phoneme Hallucinator overcomes the data scarcity bottleneck of one-shot VC. It fills the gaps in the target speaker's feature space, ensuring robust and intelligible conversion from minimal reference audio. Future work includes refining vocoder performance on hallucinated data and exploring cross-lingual generalization.
Summary updated