Translate the Sources, Not the Mixture: Learning Realistic Training Mixtures from Unpaired Recordings

Separating a recording into its individual voices or instruments works well when a model is trained on mixtures paired with the clean isolated tracks, often called "stems." Such pairs almost never exist for real rooms, live bands or meetings. The standard workaround is to add clean recordings together in software. Models trained that way collapse on real audio: REAL-M reports separators that reach about 20 dB SI-SNR (a scale-invariant signal-to-noise ratio, higher is better) on synthetic mixtures but only about 2.8 dB on crowd-sourced real recordings. Unpaired translation offers a way to make synthetic data look real. Unpaired translation means learning to map one domain onto another, such as simulated scans to real scans, from two unmatched collections and no matched pairs. The obvious version translates the synthetic mixture into a realistic one. That breaks the very thing synthetic data was for: once the mixture is altered by a free-form generator, the original clean stems no longer describe it.

This note proposes the opposite placement. The model passes each clean source through a learned, physically shaped acoustic channel: per-source room filters, gains, shared background noise and a shared mild post-mix distortion. It then sums the results. A discriminator, a classifier that tries to tell generated from real, sees only the summed mixture and compares it against unlabeled real mixtures from the target environment. Because the translation happens per source and before the sum, every generated mixture comes with exact stems by construction. Because the channel is a constrained signal-processing model rather than a free image generator, it cannot invent phonemes or hide information. The question is how much of the gap between unsupervised separation and a target-trained "oracle" this closes, measured where an oracle can actually be computed.

Core Claim

In linear acoustics, a real mixture is a sum of per-source "wet" signals: each source convolved with its own path to the microphone. Noise and mild device nonlinearities are added on top. The failure documented by REAL-M is therefore not that superposition is false. It is that the summed signals are dry and wrong for the room. We write the generator as

$$\tilde m = g_\psi\Big(\sum_{i=1}^{K} h_\theta(e, i) * s_i + n_\theta(e)\Big), \qquad e \sim p(e).$$

Here $s_i$ are clean source recordings from a solo corpus. $e$ is a random environment code shared by all $K$ sources in a mixture. $h_\theta(e,i)$ is a generated impulse response for source $i$, and $*$ is convolution. $n_\theta(e)$ is generated background noise. $g_\psi$ is a small, near-identity post-mix nonlinearity standing in for microphone and codec effects. The parameters $\theta,\psi$ are trained adversarially so that $\tilde m$ is indistinguishable from real target mixtures, with no target stems.

The training targets for a separator are $y_i = h_\theta(e,i) * s_i$, the wet stems, or $s_i$ when dereverberation is wanted. These targets are exact up to the small nonlinearity $g_\psi$. Our claim is specific. This additivity-preserving, structurally restricted, one-directional translator closes substantially more of the oracle gap than three alternatives:

  • randomized physical simulation with the same filter family;
  • unconstrained mixture-level CycleGAN translation;
  • unsupervised remixing trained directly on the target mixtures.

Where This Sits in Prior Work

The motivation comes from the reality-gap literature. REAL-M measures the collapse on real mixtures. Low-Frequency Compensated Synthetic Impulse Responses shows that ray-tracing room simulators miss low-frequency wave behaviour. Slakh shows that synthetic music data helps only when rendered at high fidelity: bass SI-SDR moved from -0.5 dB to 1.3 dB with Slakh2100, but only to 0.0 dB with the cheaper Flakh renders. Together these say that realism of the rendering matters and hand-built simulators do not supply enough of it. We fit the rendering to real recordings instead of hand-designing it.

The method borrows from unpaired translation but changes its placement and form. A Novel Domain Adaptation Framework for Medical Image Segmentation is the closest analogue: it renders synthetic tumours with a physics model, translates them to realistic MRI with CycleGAN, and keeps the synthetic labels. That paper notes CycleGAN can move lesion boundaries, which is exactly the label corruption we avoid. CycleGAN-based Unpaired Speech Dereverberation shows unpaired spectrogram translation works for one speaker. It also found sample-level SDR favoured paired training, consistent with free-form spectrogram generators failing to keep phase. The file also cites analyses of CycleGAN hiding source information in near-invisible high-frequency patterns ("steganography"). UEGAN drops the backward generator and instead relies on a VGG content loss trained on photos, which has no audio counterpart. Our answer to the same problem is structural. A short convolution, a gain and additive noise cannot fabricate content, and with no backward generator there is no reconstruction loss to cheat.

Two lines of work constrain the design. Adversarial UDA for Harmonic-Percussive Separation aligns internal features of the separator across domains. It reached 7.4 dB percussive SDR on Tap & Fiddle from 1.3 dB, but left an oracle gap. Conditional GAN for Structured Domain Adaptation argues that global feature alignment blurs structured outputs. Both push us toward the input level.

The strongest competitor is SURF. It trains a flow-matching separator purely from mixtures by remixing a teacher's estimates, reaching 16.54 dB on Libri2Mix against 17.89 dB supervised. It was evaluated only where training and test mixtures share a distribution, and its pseudo-stems inherit the teacher's errors. Our method instead gets exact stems but must model the environment. The two are complementary, and the experiments test both alone and combined.

Research Program

The translator. We build the translator in three parts:

  • A small network maps the environment code $e$ and a source index to an impulse response of up to about one second. This is parameterized as a direct-path delay and gain, sparse early reflections, and a decaying late tail with frequency-dependent decay.
  • A noise generator produces stationary and slowly varying background.
  • A low-capacity post-mix block (a few gated filters with a soft clipper) models the device.

The discriminator is multi-resolution and operates on complex spectrograms of the summed mixture, so phase and reverberation tails are visible to it.

To stop the generator collapsing to a single room, we add a diversity term between pairs of environment codes. We also regularize $g_\psi$ toward identity, so that the wet-stem targets stay accurate. The mismatch $\|g_\psi(\sum_i y_i) - \sum_i g_\psi(y_i)\|$ is logged throughout training as a direct measure of target error.

The analysis. One deliverable is a measurement of the oracle gap, alongside the separator itself. In every setting with ground-truth target stems, we report three quantities: the source-only result, the adapted result, and the oracle trained on target stems. The headline number is the fraction of the gap recovered. The same scheme is applied to every method, so the comparison does not hinge on raw dB.

Hybrid variant. We also test using translated mixtures to pretrain or anchor a SURF-style remixing student on the real target mixtures. Exact synthetic stems may reduce the confirmation bias that DUST documents for self-training under shift.

Baselines

Source-only (the do-nothing reference). A separator trained on plain summed mixtures: LibriMix-style for speech, per LibriMix, and Slakh2100 or MUSDB18 for music. On REAL-M, the reported reference is SepFormer at about 2.8 dB SI-SNR and 60.7% downstream WER (REAL-M). We retrain it with our separator backbone so that only the training data differs.

Randomized simulation. The same impulse-response, noise and post-mix family, but with parameters drawn from broad hand-set priors instead of fitted adversarially. This isolates the key ingredient: whether fitting the channel to real mixtures matters beyond having a physically shaped channel at all. Beating it requires a gain in gap fraction at a matched number of generated mixtures and matched separator training steps.

Mixture-level translation. A CycleGAN, in the style of the dereverberation work, that maps synthetic mixtures to the real target collection. The separator is then trained against the original stems. It receives the same unpaired real data and a comparable generator parameter budget. A second ablation, an unconstrained per-source generator, separates the effect of "per-source placement" from that of "structural restriction."

Feature-level adversarial alignment, following Adversarial UDA for HPSS. Its reported 7.4 dB percussive SDR on Tap & Fiddle is the number to beat in that setting.

SURF trained directly on the unlabeled target mixtures. This is the strongest existing method for learning without stems. Its reported results are 16.54 dB SI-SDR on Libri2Mix, versus 12.39 dB for MixIT and 17.89 dB supervised (SURF). No reported result on Lacuna was found for SURF on REAL-M, Tap & Fiddle or MUSDB18, so we will run it ourselves. It gets the same target mixtures, the same backbone size and the same number of sweeps over a shared hyperparameter grid.

SSL two-step training. For REAL-M we also compare against the two-step SSL scheme, whose reported downstream WER is 46.25%, down from 51.10%.

Beating the field means two things. First, a higher gap fraction than both SURF-on-target and randomized simulation on at least two of the three benchmarks. Second, on REAL-M, WER below 46.25%.

Why not just…?

Why not just randomize a room simulator more aggressively? This is the most serious threat. Domain randomization often works when the randomized family covers reality. The evidence that it does not fully cover reality is indirect. Ray-traced responses miss low-frequency behaviour (Low-Frequency Compensated RIRs), and render fidelity mattered for Slakh. The randomized-simulation baseline uses exactly our filter family, so if fitting adds nothing, it will show up directly. In that case the contribution would shrink to a useful filter parameterization, and we would report it as such.

Why not just run SURF or MixIT on the real target mixtures? This needs no translation, and SURF's synthetic results are strong. Two concerns remain. Its pseudo-stems come from a teacher that starts out mismatched to the target. Remixing methods also reportedly struggle beyond three or four sources. If SURF-on-target matches us, the hybrid experiment tells us whether the exact stems are still worth adding as an anchor. If the hybrid also gives no gain, we conclude that translation is redundant whenever target mixtures are plentiful. The target-pool sweep then tells us whether it still pays off when they are scarce, for example the 50 unlabeled fiddle recordings in Tap & Fiddle.

Why not just translate mixtures with CycleGAN, as was done for medical images? Once the mixture is regenerated freely, the old stems are no longer correct targets. The cycle loss can also be satisfied by hidden high-frequency codes. The dereverberation study already saw sample-level SDR suffer under unpaired spectrogram translation. Our mixture-level baseline measures this directly. Comparing the unconstrained per-source generator with the structured one shows whether the restriction, and not just the placement, matters.

Why not just align separator features adversarially? It is cheaper. However, the HPSS result leaves a gap, and the structured-adaptation analysis suggests feature alignment blurs fine output structure. We run it as a baseline under the same data and compute.

Proposed Experiments

Music, Slakh2100 to MUSDB18. We use Slakh stems as the clean sources and MUSDB18 training mixtures as the unlabeled target, with MUSDB18 stems hidden. The oracle trains on MUSDB18 stems. The reported references are -0.5 dB bass SI-SDR for MUSDB18 alone and 1.3 dB with Slakh2100 added (Slakh). Here the post-mix block must model mastering, so the logged target-error term is especially informative.

Music, MUSDB18 to Tap & Fiddle. MUSDB18 stems are the sources and the fiddle recordings are the target. Ground-truth isolated tracks give SDR against the 7.4 dB adversarial-UDA result.

Speech, LibriSpeech-based mixtures to REAL-M. We report WER through a fixed recognizer, plus REAL-M's blind SI-SNR estimator. That estimator was validated on synthetic data (Pearson 0.82 to 0.87), so it may reward synthetic-looking output. We therefore treat WER as primary and the estimator as secondary. We also test on LRS2Mix, where the two-step paper reports 8.6 to 9.5 dB SI-SDRi, and on Libri3Mix to probe more sources.

Decisive checks. First, gap fraction against randomized simulation at matched data: this is the test of whether fitting matters. Second, comparison against SURF-on-target and the hybrid while shrinking the unlabeled target pool from all available mixtures down to a few dozen. We predict our advantage grows as the pool shrinks, because the translator has far fewer parameters than a separator. Third, an artifact check comparing structured, unconstrained per-source and mixture-level translators on sample-level SDR. If the structured translator does not beat the mixture-level one there, the anti-hallucination argument fails. Fourth, a sanity check on synthetic data. Make the "real" target Libri2Mix convolved with held-out measured room responses. Since the true wet stems are known there, we can verify that the fitted responses recover the true reverberation-time distribution and that the post-mix target error stays below roughly 1 dB SI-SDR equivalent.