Towards Multimodal Query-Based Spatial Audio Source Extraction

Changjiang Yu, Hao Ma, Li Xu, Xiao-Lei Zhang, Mingjie Shao, Chi Zhang, Xuelong LiarXiv 2025 · DOI · arXiv

Abstract

Query-based audio source extraction seeks to recover a target source from a mixture conditioned on a query. Existing approaches are largely confined to single-channel audio, leaving the spatial information in multi-channel recordings underexploited. We introduce a query-based spatial audio source extraction framework for recovering dry target signals from first-order ambisonics (FOA) mixtures. Our method accepts either an audio prompt or a text prompt as condition input, enabling flexible end-to-end extraction. The core of our proposed model lies in a tri-axial Transformer that jointly models temporal, frequency, and spatial channel dependencies. The model uses contrastive language-audio pretraining (CLAP) embeddings to enable unified audio-text conditioning via feature-wise linear modulation (FiLM). To eliminate costly annotations and improve generalization, we propose a label-free data pipeline that dynamically generates spatial mixtures and corresponding targets for training. The result of our experiment with high separation quality demonstrates the efficacy of multimodal conditioning and tri-axial modeling. This work establishes a new paradigm for high-fidelity spatial audio separation in immersive applications.

Introduction

While significant progress has been made in separating single voices from noisy backgrounds in mono or stereo recordings, spatial audio—such as the 360-degree sound used in VR/AR—remains a complex frontier. Most existing models ignore the rich directional cues present in multi-channel recordings or are limited to "fixed-class" extraction, meaning they can only isolate specific sounds they were explicitly trained on, such as speech or vehicles.

The Band-split Spatial Audio Separation Transformer (BSAST) addresses these challenges by extracting specific sound sources from First-Order Ambisonics (FOA) mixtures using either text or audio queries. This approach tackles two primary hurdles: the modeling complexity of simultaneous temporal, frequency, and spatial channel dependencies, and the scarcity of large-scale, manually annotated datasets for 3D audio source extraction.

The BSAST Framework

The core innovation of BSAST is its ability to process the "where" of a sound alongside its semantic content. The architecture is divided into several specialized modules designed to handle the high dimensionality of spatial audio.

Architecture of the Band-split Spatial Audio Separation Transformer (BSAST).

Tri-axial RoPE Transformer

Instead of a standard transformer, BSAST utilizes a "tri-axial" approach. It processes audio data sequentially along three distinct axes: Time, Frequency, and Spatial Channels. This axial attention mechanism allows the model to efficiently capture complex interactions, such as how a sound's reverberation interacts with a specific direction in a room. To maintain a sense of order across these dimensions, the model incorporates Rotary Positional Embeddings (RoPE).

Multimodal Conditioning (CLAP + FiLM)

To enable query-based extraction, the framework leverages CLAP (Contrastive Language-Audio Pretraining) embeddings. Because CLAP maps both text and audio into a shared latent space, the model can accept a text prompt (e.g., "barking dog") or an audio snippet as a query. This information is integrated into the network via FiLM (Feature-wise Linear Modulation), which dynamically shifts and scales internal features based on the query embedding.

Band-splitting for Efficiency

Processing high-resolution 3D audio is computationally expensive. BSAST mitigates this by splitting the audio spectrum into 25 non-overlapping subbands. This allows the model to process different frequency ranges—which often exhibit different physical and spatial properties—more effectively while reducing the overall computational load.

Label-Free Data Pipeline

To bypass the need for expensive human-labeled datasets, the researchers developed a label-free training pipeline. During training, the system takes the CLAP embedding of a target sound and injects controlled noise into it. This "pseudo-query" acts as a proxy for a text description, allowing the model to learn associations between semantic concepts and audio features without explicit text labels.

The training data was synthesized using high-quality dry sources (Anechoic Sound Event 1K, FSD50K, EARS) and spatial cues from the FOA-MEIR dataset. Background noise was sourced from ESC-50 and DISCO, with Spatial Scaper used to simulate realistic 3D environments with up to three overlapping sound events.

Key Results and Insights

The model was evaluated using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR), a standard metric for audio purity. The experiments yielded several critical insights:

  • Spatial Advantage: Utilizing full spatial information (WXYZ channels) achieved an SI-SDR of 7.296 dB, significantly outperforming the single-channel (W) version, which reached only 5.833 dB with audio queries. This confirms that spatial cues are fundamental to disentangling overlapping sounds in reverberant environments.
  • Multimodal Success: Despite being trained with pseudo-queries, the model performed well on real text queries, achieving an SI-SDR of 4.098 dB.
  • Scalability: Performance improved steadily as the number of Transformer blocks increased, suggesting the architecture can scale to handle even more complex acoustic scenes.
Visual comparison of the original mixture, reference sources, and estimated extractions for 'Typing' and 'Pouring' events.

As shown in the qualitative results, the model successfully isolates distinct sound events like "Typing" and "Pouring" from a complex spatial mixture, maintaining the integrity of the target waveforms.

Conclusion and Future Work

BSAST establishes a new paradigm for high-fidelity spatial audio separation. By treating text and audio as unified modalities within a tri-axial modeling framework, it offers unprecedented flexibility for immersive applications. However, a performance gap remains between audio-based and text-based queries. Future work will focus on closing this gap and improving the model's generalization to "in-the-wild" acoustic scenes that may fall outside the vocabulary of the pre-trained CLAP model.

Summary updated

Abstract

Query-based audio source extraction seeks to recover a target source from a mixture conditioned on a query. Existing approaches are largely confined to single-channel audio, leaving the spatial information in multi-channel recordings underexploited. We introduce a query-based spatial audio source extraction framework for recovering dry target signals from first-order ambisonics (FOA) mixtures. Our method accepts either an audio prompt or a text prompt as condition input, enabling flexible end-to-end extraction. The core of our proposed model lies in a tri-axial Transformer that jointly models temporal, frequency, and spatial channel dependencies. The model uses contrastive language-audio pretraining (CLAP) embeddings to enable unified audio-text conditioning via feature-wise linear modulation (FiLM). To eliminate costly annotations and improve generalization, we propose a label-free data pipeline that dynamically generates spatial mixtures and corresponding targets for training. The result of our experiment with high separation quality demonstrates the efficacy of multimodal conditioning and tri-axial modeling. This work establishes a new paradigm for high-fidelity spatial audio separation in immersive applications.
BibTeX
@misc{yu2025towards,
  author = {Changjiang Yu and Hao Ma and Li Xu and Xiao-Lei Zhang and Mingjie Shao and Chi Zhang and Xuelong Li},
  title = {{Towards Multimodal Query-Based Spatial Audio Source Extraction}},
  year = {2025},
  eprint = {2510.13308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.13308},
  howpublished = {arXiv},
}
Figure