Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion

Yu Zhang, Baotong Tian, Zhiyao DuanarXiv 2025 · DOI · arXiv

Abstract

Zero-shot online voice conversion (VC) holds significant promise for real-time communications and entertainment. However, current VC models struggle to preserve semantic fidelity under real-time constraints, deliver natural-sounding conversions, and adapt effectively to unseen speaker characteristics. To address these challenges, we introduce Conan, a chunkwise online zero-shot voice conversion model that preserves the content of the source while matching the voice timbre and styles of reference speech. Conan comprises three core components: 1) a Stream Content Extractor that leverages Emformer for low-latency streaming content encoding; 2) an Adaptive Style Encoder that extracts fine-grained stylistic features from reference speech for enhanced style adaptation; 3) a Causal Shuffle Vocoder that implements a fully causal HiFiGAN using a pixel-shuffle mechanism. Experimental evaluations demonstrate that Conan outperforms baseline models in subjective and objective metrics.

Introduction

Voice Conversion (VC) aims to transform a source speaker's identity into a target speaker's while maintaining the original linguistic content. Achieving this in a zero-shot manner—converting to speakers unseen during training—and online (real-time) presents a significant technical "trilemma":

  1. Content Preservation: High-quality extractors like HuBERT often require global context, introducing unacceptable latency for real-time use.
  2. Style vs. Timbre: Many models capture basic timbre but fail to replicate the unique prosody and rhythm (style) of the target speaker.
  3. Streaming Artifacts: Processing audio in discrete chunks often leads to spectral glitches or "checkerboard" artifacts at boundaries when using standard upsampling methods.

Methodology

Conan addresses these challenges through a modular, fully causal architecture designed for low-latency streaming. The system breaks the conversion process into three specialized components.

The overall architecture of Conan (a), detailing the Stream Content Extractor (b) and the Causal Shuffle Vocoder (c).

Stream Content Extractor

To avoid the latency of non-causal models, Conan utilizes an Emformer architecture. By distilling knowledge from a pre-trained HuBERT model, the extractor learns to produce high-quality content embeddings using only past and current audio chunks, maintained via a memory bank to preserve context without needing to see the "future."

Adaptive Style Encoder

Beyond simple timbre, Conan employs an Adaptive Style Encoder featuring Clustering Vector Quantization (CVQ). This component filters out noise to isolate fine-grained stylistic features, which are then mapped to source content chunks via an alignment attention mechanism.

Causal Shuffle Vocoder

Traditional vocoders use transposed convolutions that can cause temporal artifacts in streaming. Conan replaces these with a pixel-shuffle mechanism within a HiFiGAN framework. Pixel-shuffle reorders data rather than interpolating, ensuring strict causality and smoother transitions between audio chunks.

Experimental Evaluation

The model was trained on the LibriTTS dataset, comprising approximately 586 hours of speech. Evaluation was conducted on the VCTK dataset using speakers entirely unseen during training to validate zero-shot performance. Performance was measured using Word Error Rate (WER) for content accuracy, Resemblyzer (SIM) for speaker similarity, and Mean Opinion Scores (MOS) for subjective quality.

Results and Discussion

Conan was evaluated in two configurations: Full (80ms chunks with some lookahead) and Fast (20ms chunks, strictly causal).

Objective and subjective performance comparison of Conan against state-of-the-art baseline models.

As shown in the results table, Conan (Full) achieved a WER of 6.02%, outperforming existing streaming baselines like StreamVC. In subjective tests, Conan consistently scored higher in both Quality (MOS-Q) and Speaker Similarity (MOS-S) compared to all baselines. Notably, the "Fast" setting achieved an end-to-end latency of just 36.87ms on an NVIDIA A100, making it highly suitable for seamless real-time communication.

Conclusion

Conan demonstrates that high-fidelity, zero-shot voice conversion is achievable in real-time by combining distilled streaming content extraction with a causal, pixel-shuffle-based vocoder. While a slight performance trade-off exists when moving to ultra-low latency settings, the model sets a new benchmark for online voice conversion. Future work will focus on improving the "Fast" setting's quality and testing robustness across diverse languages and acoustic environments.

Summary updated

Abstract

Zero-shot online voice conversion (VC) holds significant promise for real-time communications and entertainment. However, current VC models struggle to preserve semantic fidelity under real-time constraints, deliver natural-sounding conversions, and adapt effectively to unseen speaker characteristics. To address these challenges, we introduce Conan, a chunkwise online zero-shot voice conversion model that preserves the content of the source while matching the voice timbre and styles of reference speech. Conan comprises three core components: 1) a Stream Content Extractor that leverages Emformer for low-latency streaming content encoding; 2) an Adaptive Style Encoder that extracts fine-grained stylistic features from reference speech for enhanced style adaptation; 3) a Causal Shuffle Vocoder that implements a fully causal HiFiGAN using a pixel-shuffle mechanism. Experimental evaluations demonstrate that Conan outperforms baseline models in subjective and objective metrics. Audio samples can be found at https://aaronz345.github.io/ConanDemo.
BibTeX
@misc{zhang2025conan,
  author = {Yu Zhang and Baotong Tian and Zhiyao Duan},
  title = {{Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion}},
  year = {2025},
  eprint = {2507.14534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.14534},
  howpublished = {arXiv},
}
Figure