To train high-fidelity audio models, machine learning systems require clean, isolated signals—such as a pure singing voice devoid of background music, or a single speaker isolated from environmental noise. Because original multi-track studio recordings are rarely available at scale, the standard practice is to use automated preprocessing tools like source separation, denoising, or acoustic echo cancellation to isolate target signals from mixed real-world audio.
However, these algorithmic filters are imperfect. They leave behind subtle digital scars: spectral holes, phase distortions, ghosting, and inter-channel bleeding. The core insight of this research direction is that downstream machine learning models do not treat these artifacts as processing errors; they treat them as valid statistical features of the target domain. This leads to a phenomenon known as artifact propagation, where the imperfections of a preprocessing tool contaminate the learning process, degrading the performance of downstream generative, classification, and recognition models.
The Anatomy of Preprocessing Distortion
Why do source separation and enhancement tools fail to deliver pristine training targets? Audio mixtures contain highly overlapping frequencies in both time and space. When an algorithm attempts to mask out an unwanted instrument or background noise, it inevitably removes parts of the desired signal or fails to suppress the interference entirely.
As noted by Yicheng Gu et al. in SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset, automated separation frequently produces "ghosting" effects, where faint, distorted remnants of the original instrumental backing remain audible in the isolated vocal track. Conversely, when isolating transient-heavy components like drums, traditional source separation introduces unnatural textures and artificial echoes, as observed by Suntae Hwang et al. in DOSE : Drum One-Shot Extraction from Music Mixture.
Compounding this issue, standard evaluation metrics can obscure the true extent of these distortions. Yuki Mitsufuji et al. point out in Music Demixing Challenge 2021 that metrics like the signal-to-distortion ratio (SDR) can produce misleading results when the ground truth reference contains microphone bleed. A model that successfully removes the bleed may be penalized for deviating from an imperfect reference, complicating the development of cleaner separation tools.
How Generative Models Memorize the Flaws
When generative models are trained on datasets derived from automated source separation, they treat the resulting digital artifacts as ground-truth characteristics of the target audio. Because deep neural networks excel at capturing high-frequency statistical regularities, they inadvertently memorize and reproduce these processing signatures.
In the context of musical accompaniment generation, Jianyi Chen et al. demonstrate in FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation that residual noises from automated separation leak directly into the synthesized results, limiting final acoustic clarity. This error propagation acts as a quality bottleneck across various generative tasks:
- Text-to-Song Synthesis: Rongjie Huang et al. find in Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment that bleeding between vocal and instrumental tracks in the training data is learned by the model, leading to degraded audio quality in the final synthesized output.
- Speech Synthesis (TTS): Chen Zhang et al. show in DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modeling that applying separate speech enhancement tools to clean training data introduces artificial distortions that the downstream synthesis model ultimately learns to mimic.
- Acoustic Echo Cancellation: Jan Franzen argues in AEC in A Netshell: on Target and Topology Choices for FCRN Acoustic Echo Cancellation that treating echo cancellation as a direct source separation task over-processes the desired speech signal, stripping away fine-grained acoustic details and introducing audible artifacts.
The Confusion of Discriminative Models
For classification, tracking, and detection models, preprocessing artifacts act as dangerous confounding variables. Instead of learning the underlying semantic or musical features, neural networks often learn to identify the non-natural processing signatures introduced by the separation tool.
Axel Roebel demonstrates in Data Augmentation for Drum Transcription with Convolutional Neural Networks that complex signal transformations introduce digital artifacts that do not exist in natural recordings. As a result, neural networks inadvertently learn to identify these processing signatures rather than the musical features, leading to poor generalization on clean, real-world data. Similarly, Ching-Yu Chiu et al. show in Source Separation-based Data Augmentation for Improved Joint Beat and Downbeat Tracking that vocal bleed and melodic remnants in isolated drum tracks introduce noise that hinders a model's ability to learn clean rhythmic representations.
This issue is particularly pronounced in tasks that rely on subtle acoustic cues, such as deepfake detection and language identification:
[Mixed Audio] ──> [Source Separation] ──> [Isolated Vocal + Artifacts] ──> [Deepfake Detector]
│
Artifacts trigger false positives ─────┘
The authors of SingFake: Singing Voice Deepfake Detection note that while source separation helps isolate vocals from background music, the processing artifacts introduced by the separator trigger false positives in deepfake detectors. This finding is echoed by Yongyi Zang et al. in SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge, where separation artifacts obscure the distinguishing features of AI-generated voices, making detection in multi-track environments significantly less reliable than in controlled settings.
Furthermore, Peter Sibbern Frederiksen et al. demonstrate in Effectiveness of Single-Channel BLSTM Enhancement for Language Identification that speech enhancement as a preprocessing step introduces artificial distortions that confuse downstream language classifiers, reducing accuracy on audio that was originally clean.
Bypassing the Preprocessing Bottleneck
To break this dependency on imperfect preprocessing, researchers are shifting toward architectures that either bypass explicit source separation entirely or explicitly account for target impurity during training.
One prominent strategy is to leverage self-supervised learning (SSL) representations that are inherently robust to background noise, eliminating the need for an initial unmixing step. Chen Wei introduces this approach in Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features, utilizing models like WavLM and HuBERT to extract clean melody features directly from dense musical mixtures without destructive source separation artifacts.
Another strategy involves redesigning optimization objectives to handle noisy targets gracefully. Matthew Maciejewski et al. address this in Training Noisy Single-Channel Speech Separation with Noisy Oracle Sources: A Large Gap and a Small Step by introducing loss functions that treat background noise as inseparable. By acknowledging that the training targets are inherently contaminated, the model avoids over-processing the signal and focuses on learning robust, invariant acoustic boundaries.