Abstract
Recently, voice conversion (VC) has been widely studied. Many VC systems use disentangle-based learning techniques to separate the speaker and the linguistic content information from a speech signal. Subsequently, they convert the voice by changing the speaker information to that of the target speaker. To prevent the speaker information from leaking into the content embeddings, previous works either reduce the dimension or quantize the content embedding as a strong information bottleneck. These mechanisms somehow hurt the synthesis quality. In this work, we propose AGAIN-VC, an innovative VC system using Activation Guidance and Adaptive Instance Normalization. AGAIN-VC is an auto-encoder-based model, comprising of a single encoder and a decoder. With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically. This one-shot VC system obtains the best performance regardless of the subjective or objective evaluations.
Introduction
Voice conversion (VC) aims to modify a speaker's identity in an audio signal while preserving the underlying linguistic content. Achieving high-quality "one-shot" conversion—where the model generalizes to unseen source and target speakers—remains a significant challenge. The primary hurdle is disentanglement: effectively separating content from speaker style.
Traditional approaches often rely on harsh bottlenecks, such as Vector Quantization (VQ) or extreme dimensionality reduction, to prevent speaker information from leaking into content embeddings. However, these methods frequently degrade audio quality, leading to robotic or muffled speech. Alternatively, redundant architectures like AdaIN-VC employ separate encoders for content and speaker information, which increases model complexity without necessarily improving disentanglement.
Methodology
AGAIN-VC introduces a streamlined auto-encoder framework that addresses these issues through two main innovations: a single-encoder architecture and Activation Guidance (AG).
Single Encoder Efficiency
Unlike previous dual-encoder models, AGAIN-VC utilizes a single encoder to extract both content and speaker information. The model leverages Instance Normalization (IN) layers within the encoder. The channel-wise mean ($\mu$) and standard deviation ($\sigma$) calculated during normalization serve as the "global style" (speaker identity), while the normalized features represent the "local content." This design eliminates the redundancy of a dedicated speaker encoder.
Activation Guidance (AG)
To prevent speaker leakage into the content embedding, the researchers introduced Activation Guidance. By applying a scaled Sigmoid function ($\alpha = 0.1$) to the content embeddings, the model creates a "soft" bottleneck. This restriction is strong enough to filter out the speaker's unique "fingerprint" but gentle enough to preserve linguistic nuances. The decoder then uses Adaptive Instance Normalization (AdaIN) to inject the target speaker's statistics into the content embedding for the final synthesis.
Experimental Setup
The model was trained and evaluated using the VCTK dataset, featuring 109 English speakers. 80 speakers were used for training, while the remaining 29 were reserved for one-shot evaluation. A pre-trained MelGAN vocoder was employed to convert the resulting mel-spectrograms back into waveforms.
Results and Discussion
AGAIN-VC demonstrates significant improvements in both efficiency and performance. The model is approximately 30% smaller than dual-encoder baselines (9.5M vs. 13.5M parameters).
Disentanglement Performance
The effectiveness of the Activation Guidance bottleneck is evident in speaker classification tests. Without guidance, speaker classification accuracy on content embeddings was approximately 68%. With the scaled Sigmoid activation, this dropped to just 1.7%, indicating near-total speaker-blindness in the content path.
As shown in the figure above, the Sigmoid activation (red line) allows for a much lower speaker classification accuracy at comparable reconstruction error levels compared to the baseline without Sigmoid (blue line). The optimal balance was found at $\alpha = 0.1$.
Subjective Evaluation
In human listening tests (Mean Opinion Score), AGAIN-VC outperformed both single-encoder (1-Enc) and dual-encoder (2-Enc) baselines in terms of naturalness and similarity to the target speaker. The "soft" bottleneck approach successfully shifted the Pareto frontier of the disentanglement-quality trade-off.
Conclusion
AGAIN-VC proves that architectural simplicity, combined with a well-tuned "soft" bottleneck, can outperform more complex voice conversion systems. By reusing internal normalization statistics for speaker identity and applying Activation Guidance to content, the model achieves superior disentanglement and audio quality with fewer parameters. Future work may explore hybrid discrete-continuous representations and cross-lingual conversion capabilities.
Summary updated