Again-VC: A One-Shot Voice Conversion Using Activation Guidance and Adaptive Instance Normalization

Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, Hung-yi Lee

Voice conversion (VC) is the "deepfake" of the audio world: changing the speaker’s identity while keeping the spoken words exactly the same. Doing this "one-shot"—where the system has never heard the source or target speaker before—is a notoriously difficult balancing act.

In a new paper, researchers from National Taiwan University introduce AGAIN-VC, a streamlined architecture that proves you can achieve better results with less hardware by rethinking how we "bottleneck" information.

TL;DR

AGAIN-VC is an elegant one-shot voice conversion system that replaces complex multi-encoder setups with a single encoder and a novel "Activation Guidance" bottleneck. By using a Sigmoid function to gently restrict content embeddings, the model achieves superior speaker disentanglement and audio quality while being 30% smaller than previous state-of-the-art models.

What Problem Does This Paper Address?

The core challenge in voice conversion is disentanglement. You want to separate what is being said (content) from who is saying it (speaker style).

Existing models usually take one of two paths, both with flaws:

  1. Harsh Bottlenecks: Methods like Vector Quantization (VQ) or extreme dimension reduction force the model to discard speaker info, but they often "break" the audio quality, making the speech sound robotic or muffled.
  2. Redundant Architectures: Models like AdaIN-VC use two separate encoders—one for content and one for speaker style. The authors argue this is redundant because the content encoder already "knows" the speaker's characteristics through its internal normalization layers.

The goal was to find a "Goldilocks" bottleneck: one strong enough to block speaker identity from leaking into the content, but soft enough to preserve the richness of the speech.

Methods

AGAIN-VC utilizes an auto-encoder framework with two key innovations:

  • The Single Encoder Efficiency: Instead of building a dedicated speaker encoder, AGAIN-VC reuses the channel-wise mean ($\mu$) and standard deviation ($\sigma$) calculated during Instance Normalization (IN) within the main encoder. These statistics represent the "global style" (the speaker), while the remaining normalized features represent the "local content."
  • Activation Guidance (AG): This is the paper's secret sauce. The researchers found that applying a specific activation function—specifically a Sigmoid with a scaling factor ($\alpha = 0.1$)—to the content embeddings acts as a highly effective information bottleneck. It constrains the range of values just enough that the model cannot hide speaker information inside the content embedding, forcing it to rely on the separate speaker path instead.
  • AdaIN Decoder: The decoder uses Adaptive Instance Normalization (AdaIN) to "inject" the target speaker's $\mu$ and $\sigma$ back into the content, effectively performing the voice swap.

Datasets

The researchers used the VCTK dataset, a standard benchmark containing 109 English speakers.

  • Training: 80 speakers (randomly selected).
  • Evaluation: The remaining 29 speakers (to test "one-shot" performance on unseen voices).
  • Vocoder: They used a pre-trained MelGAN to convert the model's output (mel-spectrograms) back into listenable waveforms.

Key Insights and Ideas

  • Less is More: The authors proved that a single encoder is "absolutely sufficient" for feature disentanglement. By extracting speaker info from the normalization layers of the content encoder, they eliminated the need for a second, redundant network.
  • The Sigmoid Advantage: Not all bottlenecks are created equal. While ReLU is too "harsh" (zeroing out half the data) and Tanh is too loose, a scaled Sigmoid provides a mild restriction that preserves linguistic nuances while effectively filtering out the speaker's "fingerprint."
  • Disentanglement vs. Quality: There is a fundamental trade-off between how well you separate speaker/content and how good the final audio sounds. AGAIN-VC shifts the Pareto frontier, achieving lower speaker leakage (measured by a classifier) without sacrificing reconstruction accuracy.

Key Results

  • Model Compression: AGAIN-VC is roughly 30% smaller than the dual-encoder baseline (9.5M parameters vs. 13.5M).
  • Superior Disentanglement: Using Activation Guidance, the speaker classification accuracy on the content embedding dropped to a mere 1.7% (down from ~68% without guidance). This means the content embedding is almost entirely "speaker-blind."
  • Subjective Quality: In human listening tests (Mean Opinion Score), AGAIN-VC outperformed both single-encoder and dual-encoder baselines in terms of naturalness and similarity to the target speaker.
  • Optimal Hyperparameters: The study found that $\alpha = 0.1$ for the Sigmoid function provided the best balance, yielding the lowest reconstruction error of all tested configurations.

Limitations and Future Work

  • Vocoder Dependency: Like most VC research, the final audio quality is heavily dependent on the vocoder (MelGAN). Any artifacts introduced by the vocoder limit the "ceiling" of the system's performance.
  • Continuous Space: While AG works well in continuous space, the authors note that discrete representations (like VQ) still have advantages for certain types of linguistic modeling, suggesting a potential hybrid approach in the future.
  • Cross-Lingual Conversion: The paper focuses on English-to-English conversion; how well this "soft bottleneck" handles the prosody of different languages remains an open question.

Summary updated

Abstract

Recently, voice conversion (VC) has been widely studied. Many VC systems use disentangle-based learning techniques to separate the speaker and the linguistic content information from a speech signal. Subsequently, they convert the voice by changing the speaker information to that of the target speaker. To prevent the speaker information from leaking into the content embeddings, previous works either reduce the dimension or quantize the content embedding as a strong information bottleneck. These mechanisms somehow hurt the synthesis quality. In this work, we propose AGAIN-VC, an innovative VC system using Activation Guidance and Adaptive Instance Normalization. AGAIN-VC is an auto-encoder-based model, comprising of a single encoder and a decoder. With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically. This one-shot VC system obtains the best performance regardless of the subjective or objective evaluations.
BibTeX
@inproceedings{chen2021again,
  author = {Yen-Hao Chen and Da-Yi Wu and Tsung-Han Wu and Hung-yi Lee},
  title = {{Again-VC: A One-Shot Voice Conversion Using Activation Guidance and Adaptive Instance Normalization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=7bF4CJ89KBu},
}

Loading related content...

Figure