Artifact Robustness in Audio Source Separation Pipelines

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2180 papers219 conceptsfrozen 2026-10-04

In the ideal world of audio engineering, source separation is a "perfect unmixing" process—taking a polyphonic song and cleanly extracting the vocals, drums, and bass into isolated tracks. In practice, however, source separation is more like trying to un-bake a cake. While you might successfully retrieve the flour, it remains inevitably dusted with sugar and stained by eggs.

These remnants are known as artifacts. As researchers move from simple unmixing to complex pipelines—where separated audio is used to train generative models or drive animations—the "artifact tax" has become a central bottleneck. Recent work focuses on understanding how these digital scars degrade downstream tasks and how to build models that can see past them.

The Isolation Paradox

The fundamental challenge in source separation is the trade-off between interference (hearing other instruments in your target track) and distortion (the target track sounding "robotic" or "watery").

Bernhard Lehner and Gerhard Widmer identified two primary failure modes: "musical noise," which consists of short-lived spectral bursts, and "ghostly remnants," where background instruments are not fully silenced but instead reduced to eerie, thin echoes.

This creates a paradox for machine learning. If a model is trained to be too aggressive in its isolation, it introduces non-linear distortions that confuse subsequent models. Chitralekha Gupta and Haizhou Li demonstrated this in lyrics transcription: they found that transcription models actually perform better on the original, messy polyphonic mix than on "cleaned" vocals that have been mangled by aggressive source separation artifacts.

The Downstream Tax

When separated audio is used as training data for generative models, the artifacts don't just disappear; they propagate.

  • Generative Replication: Giorgio Barnabò and Fabio Petroni found that generative models like CycleDRUMS, when trained on separated tracks, inadvertently learn to replicate the noise patterns of the separation algorithm itself, limiting the fidelity of the synthesized audio.
  • Animation Glitches: In the realm of audio-driven 3D head animation, Xiumei Xie noted that inaccurate voice decomposition leads to degraded lip-syncing, as the model struggles to distinguish between the primary singer's phonemes and the "spectral bleeding" of a partner's voice.
  • Voice Cloning: Vikentii Pankov observed that using external denoising to "clean" reference audio for zero-shot TTS often introduces a robotic quality, creating a direct trade-off between background noise reduction and vocal authenticity.

Shortcut Learning and Spectral Bleeding

One of the most insidious effects of artifacts is shortcut learning. In the SingSong project, Chris Donahue and colleagues discovered that high-capacity models trained on separated audio often "cheat."

Instead of learning to generate a new accompaniment based on the melody of a vocal, the model learns to listen to the nearly inaudible "bleed" (the faint traces of the original drums and guitar left in the vocal track) and simply amplifies them. This prevents the model from being truly creative, as it becomes a sophisticated "re-synthesizer" of artifacts rather than a generative musician.

Measuring and Cleaning the Mess

To solve these issues, the field is moving toward "artifact-aware" evaluation and "blind" data cleaning.

Disentangled Metrics

Traditional metrics like Signal-to-Distortion Ratio (SDR) often conflate different types of errors. Amir Ivry and Shinji Watanabe introduced MAPSS, a framework that uses manifold learning to finally disentangle perceptual separation from signal distortion. By measuring the distance between a signal and known clusters of distortions, researchers can finally diagnose why a model is failing—whether it's leaking other sources or destroying the target signal.

Blind Data Cleaning

Since manually auditing thousands of hours of training data for artifacts is impossible, Azalea Gui and Woosung Choi proposed "blind" data cleaning. Their approach uses machine unlearning and Fréchet Audio Distance (FAD) to identify and filter out contaminated samples (those with heavy bleeding or label noise) using only a tiny set of clean reference audio. This method has been shown to close the performance gap caused by "dirty" training data by nearly 67%.

Latent Integration

Rather than treating separation and generation as two separate steps, newer architectures like Ex-Diff by Yudong Yang and Qiuqiang Kong integrate them. By injecting latent features from a source separation model directly into a diffusion process, the generative model can use the "guidance" of the separation without being forced to replicate its surface-level artifacts.

Papers in this version

Showing 60 of 180 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 142 papers
  2. Version 1 · 2026-10-04 · 180 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_source_separation_artifacts_a7dbcc86,
  title = {source separation artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/source-separation-artifacts/rdr_a7dbcc86820b483e8cf03a8d87470950},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_artifact_robustness_in_audio_source_sepa_a7dbcc86_v1,
  title = {Artifact Robustness in Audio Source Separation Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/artifact-robustness-in-audio-source-separation-pipelines/rdr_a7dbcc86820b483e8cf03a8d87470950/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}