The most effective models for audio source separation—the task of isolating individual voices or instruments from a mixture—are typically trained using supervised learning. This requires "paired data": a recording of the mixture alongside perfectly clean, isolated recordings of each constituent source (often called "stems").
However, as noted by Michael Michelashvili et al., obtaining high-quality isolated tracks is difficult and labor-intensive. In many real-world scenarios, such as live musical performances or multi-talker meetings, capturing ground-truth clean audio is physically impossible. Saurjya Sarkar et al. point out that in chamber ensembles, microphone "bleed"—where one instrument’s sound leaks into another’s microphone—prevents the collection of pristine training data.
To bypass this "supervision gap," researchers have developed methods to train models using only unlabeled mixtures, weak labels, or synthetic approximations.
Synthetic Mixtures and Pseudo-Labeling
One way to overcome the lack of real-world paired data is to manufacture it. If we have access to some isolated sounds (even if they don't match our target environment), we can mix them together to create "synthetic mixtures" where the ground truth is known by design.
Michael Michelashvili et al. demonstrate this by crossing estimated voices with real instrumental tracks to create new training examples. Similarly, Max W. Y. Lam et al. propose "Mixup-Breakdown Training," a consistency method where a teacher model breaks down unlabeled mixtures into components, which are then "mixed up" again to train a student model. This enforces a logic of consistency: the model must be able to decompose a mixture into parts that, when recombined, equal the original signal.
Weak Supervision from Non-Audio Cues
When clean audio stems are unavailable, other forms of data can act as a "critic" or guide for the separation process. This is known as weak supervision.
- Visual Cues: Ruohan Gao et al. and Zhenkai Lin et al. use unlabeled video to guide audio separation. By associating visual object detections (like a violin or a person speaking) with specific frequency patterns, the models learn to isolate sounds based on what is happening on screen.
- Musical Scores: Gordon Wichern et al. propose using musical scores as supervision. Instead of needing the isolated audio of a flute, the model uses a transcription of what the flute should be playing to guide the separation.
- Speaker Characteristics: In speech extraction, Kateřina Žmolíková et al. use speaker identity (x-vectors) rather than clean references. The model learns to extract audio that matches the known "fingerprint" of a specific speaker.
Generative Priors and Architectural Bias
A more recent shift involves using the inherent structure of audio or the bias of neural networks to perform separation without any task-specific training data.
Arnon Turetzky explores the "Deep Audio Waveform Prior," showing that certain network architectures are naturally better at reconstructing clean signals than noise. By attempting to reconstruct a noisy signal, the network captures the "clean" structure first, effectively denoising the audio before it has a chance to overfit to the noise.
Generative models, particularly diffusion models, have also become powerful tools for unsupervised separation. Yulun Wu et al. and Jean-Eudes Ayilo et al. use pre-trained generative models that already "know" what clean speech sounds like. During inference, these models act as a prior, steering the separation process to ensure the output sounds like natural, clean speech, even though the model never saw the clean version of that specific noisy recording.
Domain Adaptation
Finally, researchers address the "domain mismatch" that occurs when models trained on synthetic data fail in the real world. Carlos Lordelo et al. use adversarial training to adapt models to new musical genres where no stems exist. By forcing the model to produce outputs that are indistinguishable from the distribution of the target domain, they can improve performance on niche styles (like folk music) using only unlabeled mixtures.
By moving away from a strict dependency on paired data, these approaches allow audio models to scale to the vast amount of unlabeled recordings available in the wild, making source separation viable in the messy, multi-source environments of everyday life.