Abstract
Zero-shot online voice conversion (VC) holds significant promise for real-time communications and entertainment. However, current VC models struggle to preserve semantic fidelity under real-time constraints, deliver natural-sounding conversions, and adapt effectively to unseen speaker characteristics. To address these challenges, we introduce Conan, a chunkwise online zero-shot voice conversion model that preserves the content of the source while matching the voice timbre and styles of reference speech. Conan comprises three core components: 1) a Stream Content Extractor that leverages Emformer for low-latency streaming content encoding; 2) an Adaptive Style Encoder that extracts fine-grained stylistic features from reference speech for enhanced style adaptation; 3) a Causal Shuffle Vocoder that implements a fully causal HiFiGAN using a pixel-shuffle mechanism. Experimental evaluations demonstrate that Conan outperforms baseline models in subjective and objective metrics.
Introduction
Voice Conversion (VC) aims to transform a source speaker's identity into a target speaker's while maintaining the original linguistic content. Achieving this in a zero-shot manner—converting to speakers unseen during training—and online (real-time) presents a significant technical "trilemma":
- Content Preservation: High-quality extractors like HuBERT often require global context, introducing unacceptable latency for real-time use.
- Style vs. Timbre: Many models capture basic timbre but fail to replicate the unique prosody and rhythm (style) of the target speaker.
- Streaming Artifacts: Processing audio in discrete chunks often leads to spectral glitches or "checkerboard" artifacts at boundaries when using standard upsampling methods.
Methodology
Conan addresses these challenges through a modular, fully causal architecture designed for low-latency streaming. The system breaks the conversion process into three specialized components.
Stream Content Extractor
To avoid the latency of non-causal models, Conan utilizes an Emformer architecture. By distilling knowledge from a pre-trained HuBERT model, the extractor learns to produce high-quality content embeddings using only past and current audio chunks, maintained via a memory bank to preserve context without needing to see the "future."
Adaptive Style Encoder
Beyond simple timbre, Conan employs an Adaptive Style Encoder featuring Clustering Vector Quantization (CVQ). This component filters out noise to isolate fine-grained stylistic features, which are then mapped to source content chunks via an alignment attention mechanism.
Causal Shuffle Vocoder
Traditional vocoders use transposed convolutions that can cause temporal artifacts in streaming. Conan replaces these with a pixel-shuffle mechanism within a HiFiGAN framework. Pixel-shuffle reorders data rather than interpolating, ensuring strict causality and smoother transitions between audio chunks.
Experimental Evaluation
The model was trained on the LibriTTS dataset, comprising approximately 586 hours of speech. Evaluation was conducted on the VCTK dataset using speakers entirely unseen during training to validate zero-shot performance. Performance was measured using Word Error Rate (WER) for content accuracy, Resemblyzer (SIM) for speaker similarity, and Mean Opinion Scores (MOS) for subjective quality.
Results and Discussion
Conan was evaluated in two configurations: Full (80ms chunks with some lookahead) and Fast (20ms chunks, strictly causal).
As shown in the results table, Conan (Full) achieved a WER of 6.02%, outperforming existing streaming baselines like StreamVC. In subjective tests, Conan consistently scored higher in both Quality (MOS-Q) and Speaker Similarity (MOS-S) compared to all baselines. Notably, the "Fast" setting achieved an end-to-end latency of just 36.87ms on an NVIDIA A100, making it highly suitable for seamless real-time communication.
Conclusion
Conan demonstrates that high-fidelity, zero-shot voice conversion is achievable in real-time by combining distilled streaming content extraction with a causal, pixel-shuffle-based vocoder. While a slight performance trade-off exists when moving to ultra-low latency settings, the model sets a new benchmark for online voice conversion. Future work will focus on improving the "Fast" setting's quality and testing robustness across diverse languages and acoustic environments.
Summary updated