Acoustic-Scene-Aware Target Sound Separation With Sound Embedding Refinement

Yungyeo Kim, Joon-Hyuk ChangIEEE Access 2024 · OpenReview

Abstract

Target sound separation (TSS) aims to separate specific sounds of interest, like a speech or a musical instrument, from complex acoustic environments with multiple overlapping sounds. In realistic scenarios, the important sounds that we want to hear can differ depending on transitions in the surrounding acoustic scene. This study addresses the problem of acoustic-scene-aware TSS, which separates predefined sets of target sounds considered significant for the current acoustic environment. We propose a novel approach called Acoustic-SCene-Aware Target sound separation with sound Embedding Refinement (SCATER). It refines pre-trained sound embeddings into acoustic-scene-aware representations to guide the separation of specific target sounds based on the surrounding scene. Experimental results demonstrate the superiority of SCATER over an approach that combines sound separation and scene classification separately.

Introduction

Target Sound Separation (TSS) is the "cocktail party effect" for machines—the ability to isolate a specific sound from a noisy background. However, most TSS models are static; they require a user to manually specify what to separate. In the real world, "importance" is contextual. If you are walking in a park, a bicycle bell is a critical safety signal. If you are on a busy bus, that same bell is irrelevant, but a car horn becomes vital.

Conceptual overview of acoustic-scene-aware target sound separation where the target sound changes based on the environment.

The authors identify two major hurdles in current systems: Contextual Fluidity, where target sounds change as we move through different scenes, and the Cascading Error Problem, where a single mistake by a scene classifier causes a separator to extract the entirely wrong sound, leading to total system failure.

Methods

The researchers propose SCATER (Acoustic-SCene-Aware Target sound separation with sound Embedding Refinement). The architecture consists of three main pillars:

  • MIL-based ASC: A Multiple Instance Learning-based Acoustic Scene Classification system that analyzes short audio segments, allowing the system to react almost instantly to scene transitions.
  • The Refinement Network: This core innovation takes a general embedding of all possible target sounds and "refines" it using latent information from the scene. It transforms a generic instruction into a scene-relevant cue.
  • The Separator: A backbone based on Conv-TasNet, which uses the refined embedding as a guide to mask out unwanted noise and isolate the target signals.
The detailed SCATER architecture showing the integration of the scene vector extractor, refinement network, and separator.

The authors compared several refinement techniques and found that a TCN with FiLM (Feature-wise Linear Modulation) provided the most robust results by mathematically shifting and scaling features based on scene data.

Experimental Setup and Results

To evaluate SCATER, the authors used synthetic mixtures from the TAU Urban Acoustic Scenes 2020 Mobile dataset for backgrounds and FSD50K for foreground events. They also created a custom evaluation set to test performance during mid-clip scene transitions.

Key Insights

  • Soft Cues > Hard Decisions: The refinement network acts as a buffer. Even if the scene classifier is uncertain, the refined embedding provides a "soft" cue to the separator, preventing the catastrophic performance drops seen in "hard-decision" systems.
  • Joint Optimization: By training the separator and the refinement network together, the model learns to focus on features that actually aid separation while ignoring classification noise.

Performance

In single-scene tests, SCATER achieved an overall 17.22 dB Signal-to-Distortion Ratio improvement (SDRi), nearly doubling the performance of a standard cascaded ASC+TSS combination (9.06 dB). In transition scenarios, SCATER outperformed the baseline by over 2 dB, proving its resilience as environments change.

Conclusion

SCATER represents a significant advancement in making audio processing context-aware. While currently limited by predefined target sets, the framework demonstrates that joint optimization and sound embedding refinement can overcome the traditional pitfalls of cascaded classification and separation models. Future work will explore unsupervised learning to help models discover important sounds without human labeling.

Summary updated

Abstract

Target sound separation (TSS) aims to separate specific sounds of interest, like a speech or a musical instrument, from complex acoustic environments with multiple overlapping sounds. In realistic scenarios, the important sounds that we want to hear can differ depending on transitions in the surrounding acoustic scene. This study addresses the problem of acoustic-scene-aware TSS, which separates predefined sets of target sounds considered significant for the current acoustic environment. Predefined sets of target sounds were determined beforehand based on the expected acoustic scenes. For example, the sound of a bicycle bell is predefined as the target sound in a park scene and separated from a mixture of various sounds. As a solution, we propose a novel approach called Acoustic-SCene-Aware Target sound separation with sound Embedding Refinement (SCATER). It refines pre-trained sound embeddings into acoustic-scene-aware representations to guide the separation of specific target sounds based on the surrounding scene. SCATER adopts a multiple instance learning-based acoustic scene classification system for rapid response to scene changes. The refined sound embeddings serve as cues for the TSS model, enabling the separation of different target sounds across various acoustic scenes. Experimental results demonstrate the superiority of SCATER over an approach that combines sound separation and scene classification separately.
BibTeX
@inproceedings{kim2024acoustic,
  author = {Yungyeo Kim and Joon-Hyuk Chang},
  title = {{Acoustic-Scene-Aware Target Sound Separation With Sound Embedding Refinement}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=Yn8P0NdieV},
}
Figure