Abstract
Target sound separation (TSS) aims to separate specific sounds of interest, like a speech or a musical instrument, from complex acoustic environments with multiple overlapping sounds. In realistic scenarios, the important sounds that we want to hear can differ depending on transitions in the surrounding acoustic scene. This study addresses the problem of acoustic-scene-aware TSS, which separates predefined sets of target sounds considered significant for the current acoustic environment. We propose a novel approach called Acoustic-SCene-Aware Target sound separation with sound Embedding Refinement (SCATER). It refines pre-trained sound embeddings into acoustic-scene-aware representations to guide the separation of specific target sounds based on the surrounding scene. Experimental results demonstrate the superiority of SCATER over an approach that combines sound separation and scene classification separately.
Introduction
Target Sound Separation (TSS) is the "cocktail party effect" for machines—the ability to isolate a specific sound from a noisy background. However, most TSS models are static; they require a user to manually specify what to separate. In the real world, "importance" is contextual. If you are walking in a park, a bicycle bell is a critical safety signal. If you are on a busy bus, that same bell is irrelevant, but a car horn becomes vital.
The authors identify two major hurdles in current systems: Contextual Fluidity, where target sounds change as we move through different scenes, and the Cascading Error Problem, where a single mistake by a scene classifier causes a separator to extract the entirely wrong sound, leading to total system failure.
Methods
The researchers propose SCATER (Acoustic-SCene-Aware Target sound separation with sound Embedding Refinement). The architecture consists of three main pillars:
- MIL-based ASC: A Multiple Instance Learning-based Acoustic Scene Classification system that analyzes short audio segments, allowing the system to react almost instantly to scene transitions.
- The Refinement Network: This core innovation takes a general embedding of all possible target sounds and "refines" it using latent information from the scene. It transforms a generic instruction into a scene-relevant cue.
- The Separator: A backbone based on Conv-TasNet, which uses the refined embedding as a guide to mask out unwanted noise and isolate the target signals.
The authors compared several refinement techniques and found that a TCN with FiLM (Feature-wise Linear Modulation) provided the most robust results by mathematically shifting and scaling features based on scene data.
Experimental Setup and Results
To evaluate SCATER, the authors used synthetic mixtures from the TAU Urban Acoustic Scenes 2020 Mobile dataset for backgrounds and FSD50K for foreground events. They also created a custom evaluation set to test performance during mid-clip scene transitions.
Key Insights
- Soft Cues > Hard Decisions: The refinement network acts as a buffer. Even if the scene classifier is uncertain, the refined embedding provides a "soft" cue to the separator, preventing the catastrophic performance drops seen in "hard-decision" systems.
- Joint Optimization: By training the separator and the refinement network together, the model learns to focus on features that actually aid separation while ignoring classification noise.
Performance
In single-scene tests, SCATER achieved an overall 17.22 dB Signal-to-Distortion Ratio improvement (SDRi), nearly doubling the performance of a standard cascaded ASC+TSS combination (9.06 dB). In transition scenarios, SCATER outperformed the baseline by over 2 dB, proving its resilience as environments change.
Conclusion
SCATER represents a significant advancement in making audio processing context-aware. While currently limited by predefined target sets, the framework demonstrates that joint optimization and sound embedding refinement can overcome the traditional pitfalls of cascaded classification and separation models. Future work will explore unsupervised learning to help models discover important sounds without human labeling.
Summary updated