Voice conversion systems try to keep what was said and how it was said (the melody, rhythm and emphasis, together called prosody) while swapping who says it. They work well within a gender and consistently worse across genders. In every Voice Conversion Challenge from 2016 to 2020, same-gender conversions beat cross-gender ones on naturalness and speaker similarity. Singing conversion shows the same pattern: cross-gender pairs score lower on similarity because traces of the source singer "leak" through when the model is pushed into an unfamiliar pitch register. Current systems suppress leakage with a speaker classifier attached through a gradient reversal layer, a trick that trains the encoder to make speaker labels unpredictable. That only asks the representation to hide a discrete label. It never asks the representation to stay the same when the voice itself changes continuously, for instance when the pitch register rises or the vocal tract gets shorter.
We propose to train the prosody and content encoders of a conversion system against a built-in adversary. The adversary applies the worst-case smooth, bounded change to the input voice's register and vocal tract length, and the encoder is penalized whenever its output moves. Robust training of this kind, with an inner search for the worst perturbation, normally multiplies training cost several-fold. We therefore adapt the single-step "fast" adversarial training recipe from image classification to this physiological perturbation space. The test is whether, at roughly the cost of ordinary training, worst-case warps close more of the cross-gender gap than random warps of the same kind. If they do not, the honest conclusion is that augmentation is enough, and the experiments are designed to show that either way.
Core Claim
A prosody representation should be unchanged by the things that differ between speakers of different sex: overall pitch register and vocal tract length, both of which can drift slowly within an utterance. It should still keep the fast, local pitch movements that carry intonation and emotion. This can be written as a bounded robust-optimization problem over a structured perturbation set, rather than as domain-adversarial classification.
Let $x$ be an utterance represented by its fundamental-frequency track and spectral envelope. Let $W_{a,b}(x)$ be a differentiable warp with two parts. The first, $a(t)$, adds a slowly varying offset to log-$F_0$. The second, $b(t)$, is a slowly varying frequency-warp factor that imitates a change in vocal tract length. Both curves are restricted to the span of the first $K$ cosine basis functions, which keeps them slow relative to intonation, and to a norm ball of radius $\epsilon$; we call this set $\Delta$. With $E_\theta$ the encoder, $\mathcal{L}_{\text{rec}}$ the system's usual reconstruction loss, and $\text{sg}$ a stop-gradient, we train
$$\min_\theta \; \mathbb{E}_x\Big[\mathcal{L}_{\text{rec}}(x) + \lambda \max_{(a,b)\in\Delta} \big\| E_\theta(W_{a,b}(x)) - \text{sg}(E_\theta(x)) \big\|^2\Big].$$
The inner maximization is approximated in the style of Fast is better than free, which showed that one sign-gradient step from a uniformly random start inside the allowed ball matches multi-step projected gradient descent (PGD) on CIFAR-10 and avoids "catastrophic overfitting", the failure where a model looks robust to the single-step attack but collapses under a stronger one. Here the random start is a random smooth warp, followed by one gradient step on the $2K$ basis coefficients and a projection back into $\Delta$. As a second variant we test minibatch replay from Adversarial training for free!, which reuses a single backward pass to update both the weights and the perturbation.
The new element is the perturbation space: continuous, low-frequency, grounded in physiology, and not the pixel-level noise these methods were built for. Because it is structured, the adversary is searching over speaker physiology, not over imperceptible noise.
Relation to Prior Work
Four lines of work motivate the problem. The challenge analyses establish the cross-gender gap: Analysis of the VCC 2016 results, VCC 2018 and VCC 2020. LDM-SVC attributes lower cross-gender singer similarity to timbre leakage in unnatural registers. The asymmetry reported by Yang et al. and Erro et al., with male-to-female conversion degrading more, suggests the leakage is directional.
Three systems use the mechanism we want to replace. M3 attaches a gradient-reversal speaker classifier to a phoneme-level style encoder and still names cross-gender transfer as unsolved. Kim and Chang's cross-lingual prosody transfer applies gradient reversal to speaker, language and prosody embeddings. ZSVC needs a vector-quantized bottleneck, a timbre-blurring normalization and gradient reversal together, trained on 44,000 hours. All three penalize whether a categorical label can be predicted, and none bounds how the representation responds to a continuous change in the voice. We keep their architectures but add the bounded invariance term. ZSVC's timbre blurring is the closest ingredient in spirit, since it randomly perturbs a timbre embedding. It is random rather than worst-case, however, and it operates on an embedding rather than on the acoustic physiology.
DiffProsody uses "adversarial" in the generative-model sense: a discriminator judges whether prosody and spectrogram match, and the method's efficiency gain is at inference, not in training. We do not compete with it; it tells us that "adversarial" in speech has so far never meant robust optimization.
On Prosody Modeling for ASR+TTS based VC constrains the design. It shows that copying source prosody causes perceptual speaker mismatch, and that predicting prosody from text avoids the mismatch by discarding source expression. Our method must keep source expression, which its authors' results suggest is where the difficulty lies.
Two papers supply the method: Fast AT and Free AT. Free AT also constrains it, because its authors report that clean accuracy drops as the replay factor grows. That matters here because conversion datasets are small, around 5 minutes per speaker in VCC2020. Robustness Tokens optimizes a few input tokens on a frozen vision transformer with an objective that keeps clean features and corrects attacked ones. Our invariance term borrows that idea. We also test a variant with a frozen encoder plus a few learned prefix tokens, to see whether the robustness can be added cheaply after pretraining.
Research Program
We first build the differentiable warp operator. It acts on a source-filter representation, with an $F_0$ track and a spectral envelope warped along frequency, and is resynthesized to the mel features each system consumes, so gradients can flow from the encoder back to the warp coefficients. We check that random warps sampled at the chosen radius sound like plausible other speakers rather than like artifacts. A listening check on a small sample decides the radius and the number of basis functions $K$; we start with $K$ small and a log-$F_0$ radius that covers roughly the typical male-female register difference, which is a design choice and not a figure from the literature.
We then insert the invariance term into two host systems. The first is a speech voice-conversion model with a separate prosody encoder, trained on VCTK. The second is a So-VITS-SVC-style singing model, whose latent space LDM-SVC builds on, trained on OpenSinger. For each host we train five versions. The first is the host with its original gradient-reversal classifier and no invariance term. The second adds invariance trained with random warps only. The third uses 10-step PGD warps. The fourth uses fast single-step warps with random start. The fifth uses free replay with replay factor $m$ swept.
The analysis has three parts. The first is the cost-robustness curve: measured GPU-hours against leakage. The second checks whether catastrophic overfitting appears in this low-dimensional smooth space, by attacking each trained encoder with a 20-step warp adversary. The third tests Gap 4 from the research file directly. We inspect the warps the adversary actually finds, grouped by source gender, and sweep separate radii for upward and downward register shifts, asking whether the male-to-female asymmetry shows up as an asymmetric worst case.
Baselines
The naive reference point is each host system trained exactly as published, including its gradient-reversal speaker classifier, with no warp term. Every other version must beat it on the cross-gender gap without losing same-gender quality.
The strongest existing disentanglement competitor is the combination that ZSVC needed: a narrow vector-quantized bottleneck plus random timbre perturbation plus gradient reversal. We reimplement it inside the same host, with the same data and the same hyperparameter-tuning budget. We cannot reuse ZSVC's reported numbers, a style-similarity MOS of 3.96 and naturalness of 3.86, because they come from a 44,000-hour model evaluated on a different task. We also found no reported result on Lacuna for cross-gender-specific scores from ZSVC or LDM-SVC, which reports only that cross-gender SMOS is systematically lower. So all comparisons are run by us under matched conditions.
As a fixed external reference for intelligibility on VCC2020 Task 1, Huang et al. report 13.1% word error rate (WER) when predicting prosody from text and 14.6% when transferring source prosody, against a 24.0% baseline. The text-prediction system is our reference for mismatch-free but expression-free conversion.
The key ablation is random warp augmentation over the identical perturbation set, radius and basis, run for the same number of optimizer steps and the same wall-clock budget as the fast adversarial version. This isolates whether the worst-case search, rather than exposure to warps, does the work. Multi-step PGD serves as the robustness ceiling and cost reference.
Beating the strongest competitor means a statistically significant reduction in the cross-gender minus same-gender gap in speaker similarity, and in probe-measured leakage, with source-prosody preservation no worse. Beating the key ablation means the same, at equal compute, against random warps. For cost, fast warps should come within the tuning noise of PGD robustness at a small fraction of its measured training time.
Why not just…?
Why not just normalize pitch and vocal tract length with signal processing before encoding? Mean-variance $F_0$ matching and classical vocal tract length normalization are the obvious zero-training fix. The challenge results show that systems of this era clustered between 2.0 and 3.5 MOS in VCC 2016. Liu et al. report that cross-register excitation conversion is unstable, because vocal fold vibration and tract resonances interact nonlinearly. Normalization also removes only a global offset and misses slow within-utterance drift. We still include input normalization as a cheap preprocessing baseline, because it is inexpensive to run and a reviewer will ask.
Why not just augment with random pitch and formant shifts? This is the real threat. In a space with only $2K$ dimensions, random sampling may cover the worst cases well enough, and the vision result that random noise alone fails (Fast AT) comes from a far higher-dimensional space and may not transfer. We therefore make the equal-compute random-warp ablation the central experiment and sweep $K$. Our prediction is that the adversarial advantage grows with $K$. If random warps match adversarial ones at every $K$ we will report that, and the contribution becomes a cheap, well-specified augmentation recipe rather than a robustness method.
Why not just predict prosody from text? Huang et al. show that this removes mismatch and lowers WER. It does so by discarding the source's rhythm, emotion and melody, and it is inapplicable to singing, where the melody must be transferred. We report it as the intelligibility and naturalness reference, and we measure source-contour correlation to show what it gives up.
Why not just use a tighter bottleneck or a stronger speaker classifier? ZSVC found its bottleneck insufficient on its own. Huang et al. found that bottleneck and quantized features still leak pitch and identity. A classifier-based penalty can also hide identity from its own classifier while leaving it recoverable by a different one. Our nonlinear leakage probe, trained after the fact on frozen representations, measures exactly this.
Why worry about cost at all; why not just run PGD? For a small encoder, 10-step PGD may be affordable, and we will report its measured cost honestly. The cost argument matters when the encoder is a large pretrained model, or in the end-to-end variant where the adversary's loss runs through the decoder. There, each inner step adds a full forward and backward pass through the whole system, and Free AT reports that multi-step adversarial training costs 7 to 30 times more than standard training on images.
Proposed Experiments and Decisive Checks
Training and evaluation use VCTK and OpenSinger. VCC2020 Task 1 is used for external comparability on intelligibility. ESD is used to test emotional prosody, where Hu et al. identify cross-physiology emotion transfer as unsolved. Every result is split into four directions: male-to-male, female-to-female, male-to-female and female-to-male.
The metrics cover five things. Speaker similarity is measured with an objective embedding similarity and a subjective similarity MOS. Leakage is the accuracy of a two-layer MLP probe predicting source speaker and source gender from the frozen prosody representation. Source expression preservation is the correlation and RMSE between source and output $F_0$ contours after register normalization. Quality is naturalness MOS and WER. Training cost is measured GPU-hours.
The first decisive check is the equal-compute comparison between fast adversarial warps and random warps on the cross-gender similarity gap and on probe leakage, across $K$. The second is whether fast and free training come within tuning noise of PGD robustness under a 20-step warp attack, without collapse. The third is whether the invariance term reduces the male-to-female deficit specifically, and whether that deficit's reduction depends on an asymmetric radius. The fourth is whether source-contour correlation stays at the level of the naive host, which rules out the trivial solution of flattening prosody to achieve invariance.
A positive outcome on the first and fourth checks would show that cross-gender leakage is partly a failure of worst-case invariance and can be fixed at near-standard training cost. A negative first check would redirect the field toward cheap structured augmentation.
References
- Fast is better than free: Revisiting adversarial training
- Adversarial training for free!
- Analysis of the Voice Conversion Challenge 2016 Evaluation Results
- The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
- Voice Conversion Challenge 2020 -- Intra-lingual semi-parallel and cross-lingual voice conversion --
- LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance
- Cross-Lingual Voice Conversion with Disentangled Universal Linguistic Representations
- ML Parameter Generation with a Reformulated MGE Training Criterion — Participation in the Voice Conversion Challenge 2016
- Incorporating Cross-Speaker Style Transfer for Multi-Language Text-to-Speech
- Adversarial and Sequential Training for Cross-lingual Prosody Transfer TTS
- ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training
- DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training
- On Prosody Modeling for ASR+TTS Based Voice Conversion
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- WaveNet Vocoder with Limited Training Data for Voice Conversion
- DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech - A Study Between English and Mandarin