Cycle Consistency That Cannot Hide: Phase-Scrambled Waveform Translation

Many audio tasks have no paired training data. Nobody has recordings of the same Bach passage played by the same performer on piano and on flute. Nobody has the same sentence recorded both in a reverberant hall and in a dry booth. The standard answer, borrowed from unpaired image translation, is cycle consistency. You train one network to map A to B and a second to map B back to A, then require that the round trip returns the input. In audio, every such system so far runs this cycle on a picture of the sound, a spectrogram or cepstrum. It then hands the result to a separately trained vocoder, the network that turns a spectrogram back into a waveform. We propose running the cycle directly on raw waveforms and arguing openly about what the cycle loss should compare.

The central observation is that a waveform-level cycle creates a serious "hiding" problem. Cycle-trained generators are known to smuggle source information into imperceptible details of their output so the reverse generator can recover it, rather than performing a real transformation. A raw waveform offers an enormous hiding channel: its phase and sub-audible components, which listeners and most discriminators ignore. Our claim is that a waveform cycle becomes trustworthy only if the round trip passes through a random perceptually-invisible perturbation. Concretely, the translated audio is phase-scrambled by a random all-pass filter before the reverse generator sees it, and the loss compares the round trip only on magnitude spectra at several resolutions. Any information hidden where listeners cannot hear it is then destroyed, and the cycle can only be satisfied by information the translation actually makes audible. We further claim that this mechanism is what separates a waveform cycle from the obvious alternative: a spectrogram cycle followed by a strong vocoder.

What exists and what is missing

Unpaired audio translation has so far been two-stage.

  • TimbreTron runs a CycleGAN on constant-Q spectrograms and synthesizes audio with a separately trained WaveNet. It reports lingering problems with attack transients and loudness drift.
  • CycleGAN-VC and CycleGAN-VC3 convert mel-cepstra or mel-spectrograms and rely on WORLD or a frozen MelGAN.
  • CyclePPG-XVC and MulliVC put cycles on phonetic or timbre representations but still decode to mel-spectrograms.
  • CycleGAN-based Unpaired Speech Dereverberation works on spectrograms.
  • UnSE uses optimal transport with complex spectrogram masks instead of cycles.

All of these motivate the proposal: each leaves the waveform outside the loop.

On the other side, direct waveform generators supply our method but not our training signal:

  • NVC-Net converts voices end-to-end on raw audio with multi-resolution waveform discriminators. It separates content from speaker using encoder bottlenecks, not cycles.
  • BigVGAN supplies the generator design we reuse: periodic Snake activations and anti-aliased blocks. It also supplies the loss recipe (multi-resolution spectral loss plus multi-period and multi-scale adversarial losses), but only in paired reconstruction.

Two papers constrain the design:

  • The dereverberation CycleGAN paper found that unpaired models matched paired ones on frame-level spectral metrics but lost badly on sample-level signal-to-distortion ratio, because nothing rewards sample alignment. We take this as a sign that sample-level cycle losses are the wrong target, not something to repair.
  • On The Classification-Distortion-Perception Tradeoff shows that restoration cannot raise semantic accuracy above what the degraded signal preserves. The mirror image is our concern: a cycle score can look excellent while carrying information the listener never hears. This is exactly the failure we want to measure and prevent.

Core Claim

Let $G_{AB}$ and $G_{BA}$ be waveform-to-waveform generators. Let $a$ be a waveform from domain A. Let $P$ be a random perturbation drawn from a family $\Pi$ of transformations that listeners perceive as identity: random all-pass (phase-only) filters, sub-millisecond time shifts, and low-level noise shaped below a fixed spectral masking curve. Let $S_r$ be the log-magnitude short-time Fourier transform at resolution $r$, drawn from a set $R$ of window sizes. The cycle loss is

$$ L_{cyc} = \mathbb{E}_{a}\,\mathbb{E}_{P \sim \Pi} \sum_{r \in R} \big\| S_r\big(G_{BA}(P(G_{AB}(a)))\big) - S_r(a) \big\|_1 . $$

The loss has two parts, and each closes a different loophole:

  • Comparing magnitudes rather than samples means the reverse generator is never asked to reproduce the input's phase. This removes the pathology the dereverberation paper documents.
  • Perturbing the intermediate signal means the forward generator cannot pass information to the reverse generator through phase or masked components, because $P$ randomizes those.

The full objective adds three terms:

  • multi-period and multi-scale waveform adversarial losses and discriminator feature matching per domain, as in BigVGAN;
  • a magnitude-domain identity loss, meaning $G_{AB}$ applied to audio already in B should leave it unchanged;
  • the symmetric cycle loss for B.

Semantic losses from pretrained recognizers are deliberately left out of the core method. The target regimes, instruments and room acoustics, have no mature recognizers. That absence is precisely why unpaired cycles are needed there rather than a phonetic bottleneck.

Research Program

We build a waveform CycleGAN with BigVGAN-style generators, sized to roughly match a CycleGAN-VC3 translator plus vocoder in parameters. We implement the perturbation family $\Pi$ as a differentiable layer, so gradients pass through it.

The analysis centers on a measurement we call the hidden-information gap, which can be applied to any trained cycle system, including baselines. Take a forward translation $G_{AB}(a)$ and create a perceptually equivalent copy. Three versions are used: the phase-scrambled copy, a lossy-compressed copy, and a copy re-synthesized from its mel-spectrogram by a frozen BigVGAN. Feed both the original translation and the copy to $G_{BA}$. The gap is the increase in spectral cycle error on the copy. A large gap means the cycle was being satisfied through channels a listener cannot hear.

We expect the gap to be largest where the source domain carries information the target lacks. The clearest case is dereverberation. The dry target has no room, yet the reverse map must re-create the specific room of the original reverberant input. This is the textbook setting for hiding, and it makes dereverberation the sharpest test.

We then study the tradeoff between perturbation strength and translation quality. The perturbation should remove hiding without destroying legitimate content. Content that survives realistic compression and phase randomization is, by construction, content a listener can hear.

Proposed Experiments

We use three regimes from the file.

Musical timbre transfer. This follows TimbreTron's setup on its MIDI-rendered Bach and Chopin and YouTube data. It is the primary regime because no content recognizer exists, and it stresses attack transients, where TimbreTron reports weakness. The reference point is TimbreTron's listening test: listeners judged the output similar to the target instrument in 71.7% of cases, and judged the musical piece preserved in 67.5% of cases. We will rerun that same listening protocol on all systems.

Unpaired dereverberation. This uses the Librivox and AMI setup of the dereverberation CycleGAN paper. We report frame-level spectral metrics and signal-to-distortion ratio. We also report the hidden-information gap, the variable we most expect to move.

Non-parallel voice conversion. This uses the VCC 2018 Spoke task. The Voice Conversion Challenge 2018 reports a mean opinion score (a 1-to-5 naturalness rating) of 4.1 for the winning WaveNet-based team and 3.2 for the GMM baseline, against 4.5 for natural speech. No exact CycleGAN-VC3 scores are reported on Lacuna. We therefore rerun it ourselves and report naturalness, speaker similarity, and mel-cepstral distortion. We also report word error rate from an off-the-shelf recognizer, used only for evaluation.

Baselines

Naive reference. The same waveform generators trained with a sample-level L1 cycle loss, as in image CycleGAN, with no perturbation. This is the do-nothing transfer of the image recipe. We expect it either to fail to converge to good audio or to show a large hidden-information gap. Beating it on both listening scores and the gap is the minimum bar.

Strongest existing method. A two-stage system: CycleGAN-VC3 translating mel-spectrograms, followed by the public NVIDIA BigVGAN weights. To make the comparison fair, we give this pipeline a remedy for its known weakness. The vocoder is fine-tuned on the translator's own outputs for timbre and voice, and on dereverberated mels for the room task, using the same training data. It receives the same GPU-hour budget and the same number of hyperparameter trials as our model. This is the strongest competitor because BigVGAN already handles phase and aliasing well, and fine-tuning removes most of the training-inference mismatch that VISinger and DurIAN-E 2 identify.

We count as beating it any of three outcomes:

  • a statistically significant listening-test preference on timbre transfer, especially on percussive and plucked material;
  • equal quality with a lower hidden-information gap on dereverberation;
  • matched naturalness at lower total inference cost.

Parity on everything would mean the waveform cycle is not worth its complexity. We would report that as the finding.

Ablations that isolate the key ingredient. From the full model we remove, one at a time:

  • the perturbation $P$, keeping the magnitude cycle;
  • the magnitude comparison, keeping $P$ but returning to a waveform L1 cycle;
  • the all-pass component of $\Pi$, keeping only noise.

The last ablation tests whether phase scrambling specifically, rather than generic noise, is what matters.

Why not just…?

Why not just translate mel-spectrograms and use a fine-tuned BigVGAN? This is the real threat, and the proposal does not assume it loses. The evidence against it is indirect. CycleGAN-VC3 needed special normalization modules just to keep time-frequency structure intact. TimbreTron's two-stage pipeline still smeared attacks and drifted in loudness. The fine-tuned two-stage baseline above settles the question directly. The hidden-information gap also reveals whether the mel cycle is quietly hiding information in fine spectral detail that the vocoder happens to carry.

Why not just add Gaussian noise to the translated signal, the usual fix for hiding in image CycleGANs? White noise loud enough to destroy hidden phase information would be clearly audible. Noise quiet enough to be inaudible leaves the phase channel untouched. All-pass filtering randomizes phase at essentially no perceptual cost, which is why it is the audio-specific choice. The noise-only ablation tests this claim directly. If noise alone closes the gap equally well, the contribution shrinks to "use cycle perturbation on waveforms," and we will say so.

Why not just skip cycles and use pretrained content features with speaker embeddings, as in NVC-Net? For speech this is a strong alternative. CyclePPG-XVC reports that cycle constraints on linguistic representations cut cross-lingual word error rate from 31.07% to 13.50%, which suggests bottlenecks alone leak. NVC-Net itself reports weak prosody control. For instruments and room acoustics, however, no such content encoders exist. That is why timbre transfer and dereverberation, not voice conversion, are the primary regimes. On VCC 2018 we treat a bottleneck system as a competitor rather than claiming superiority in advance.

Decisive Checks

The idea stands or falls on two results.

The first concerns the waveform cycles themselves. Without the perturbation, waveform cycle systems should show a large hidden-information gap on dereverberation, and the perturbation should shrink that gap without lowering listening scores. If there is no gap to begin with, the premise is wrong.

The second concerns the comparison with the fine-tuned mel-plus-BigVGAN pipeline. On timbre transfer, the perturbed waveform cycle must at least match that pipeline in the TimbreTron-style listening test and win on transient-heavy material. If both checks pass, the result would show that cycle translation can move into the waveform domain, provided the cycle is scored only on what listeners can hear.