MC-LExt: Multi-Channel Target Speaker Extraction with Onset-Prompted Speaker Conditioning Mechanism

Tongtao Ling, S. He, Pengjie Shen, Zhong-Qiu WangarXiv 2025 · DOI · arXiv

This paper introduces MC-LExt, a novel framework for Multi-Channel Target Speaker Extraction (MC-TSE) that moves away from complex spatial modeling in favor of a surprisingly simple "prompting" mechanism.

TL;DR

MC-LExt extracts a specific speaker's voice from a multi-microphone mixture by simply prepending a short "enrollment" clip of that speaker to the beginning of the audio. This "onset-prompting" allows the neural network to jointly learn spatial and identity cues in a fully end-to-end manner, significantly outperforming traditional methods in noisy and reverberant environments.

What Problem Does This Paper Address?

Target Speaker Extraction (TSE) is the "cocktail party problem" with a twist: you have a recording of multiple people talking, and you want to isolate just one specific person based on a "clue" (like a pre-recorded snippet of their voice).

In multi-channel (multi-microphone) setups, existing solutions usually fall into two camps:

  1. DOA-based: These estimate the "Direction of Arrival" to physically locate the speaker. They are notoriously sensitive to microphone geometry and struggle when speakers are standing close together.
  2. Embedding-based: These use a pre-trained model to turn the enrollment clip into a single fixed vector (a "fingerprint"). While useful, these vectors are "global" and lack the fine-grained temporal and spectral details needed to distinguish speakers in messy, echoing rooms.

The authors aim to create a system that is robust to noise and reverberation without requiring explicit knowledge of microphone positions or complex spatial math.

Methods

The core innovation is Multi-Channel Listen to Extract (MC-LExt). Instead of processing the enrollment clip separately, the authors prepend it to every channel of the multi-microphone mixture.

Key Technical Components:

  • Onset-Prompting: By sticking the enrollment clip at the start, the model treats it as the "earliest speech onset." This triggers the network to "look" for that specific voice and its spatial characteristics within the subsequent mixture.
  • Efficiency Tricks: Prepending audio makes the input longer and more computationally expensive. To fix this, the authors introduced:
  • Downsampling: A lightweight module that compresses the enrollment segment's time resolution.
  • Selective Blocks: Only the first few layers of the deep neural network (DNN) process the enrollment segment. The later layers focus exclusively on the mixture, significantly reducing the "GMACs" (computational load).
  • Hybrid Conditioning: They found that combining the onset prompt with a traditional global speaker embedding (fused via multiplication) yielded the best results.
  • Negative Pair Training: To prevent the model from "hallucinating" speech when the target speaker isn't actually in the room, they used a Log-MSE loss function. This encourages the model to output absolute silence if the prompt doesn't match anyone in the mixture.

Datasets

The model was evaluated on two challenging, noisy, and reverberant benchmarks:

  • WHAMR!: A 2-channel dataset derived from the Wall Street Journal corpus, featuring significant background noise and room echoes.
  • MC-Libri2Mix: A 4-channel extension of the popular Libri2Mix dataset, simulating complex multi-speaker environments.

Key Insights and Ideas

  • Spatial-Identity Synergy: By placing the prompt in the same signal path as the mixture, the DNN naturally learns the spatial correlation between the prompt and the target. It doesn't need to be told where the microphones are; it "hears" the relationship.
  • More than a Vector: The results suggest that the raw audio prompt provides much richer guidance than a compressed speaker embedding alone, especially for handling frame-level details in reverberant spaces.
  • Less is More (Processing): Interestingly, processing the enrollment segment through fewer blocks (Selective Blocks) actually improved performance in some cases. The authors suggest that over-processing the prompt might introduce noise or redundancy that degrades extraction accuracy.

Key Results

  • State-of-the-Art Performance: On the WHAMR! dataset, MC-LExt achieved a 19.1 dB SI-SDRi (Signal-to-Distortion Ratio improvement), outperforming the monaural version (17.1 dB) and standard embedding-based TSE (17.4 dB).
  • Superior to DOA Methods: On the 4-channel MC-Libri2Mix, MC-LExt achieved 16.3 dB SI-SDRi, more than doubling the performance of the strong "L-SpEx" baseline (7.4 dB).
  • Robust Silence: Thanks to the Log-MSE loss, the model achieved an energy suppression ratio of 61.0 dB on negative pairs, meaning it is exceptionally good at staying quiet when the target speaker is absent.

Limitations and Future Work

  • Computational Overhead: While the downsampling and selective blocks help, prepending audio still increases the sequence length compared to pure embedding-based methods.
  • Enrollment Length: The paper notes that longer enrollments (e.g., 30+ seconds) are common in the real world but would be computationally prohibitive for this specific prepending architecture.
  • Real-time Constraints: The study focuses on offline performance; adapting this "onset-prompting" for ultra-low-latency streaming applications remains a challenge for future research.

Summary updated

Abstract

Multi-channel target speaker extraction (MC-TSE) aims to extract a target speaker's voice from multi-speaker signals captured by multiple microphones. Existing methods often rely on auxiliary clues such as direction-of-arrival (DOA) or speaker embeddings. However, DOA-based approaches depend on explicit direction estimation and are sensitive to microphone array geometry, while methods based on speaker embeddings model speaker identity in an implicit manner and may degrade in noisy-reverberant conditions. To address these limitations, we propose multi-channel listen to extract (MC-LExt), a simple but highly-effective framework for MC-TSE. Our key idea is to prepend a short enrollment utterance of the target speaker to each channel of the multi-channel mixture, providing an onset-prompted conditioning signal that can guide TSE. This design allows the deep neural network (DNN) to learn spatial and speaker identity cues jointly in a fully end-to-end manner. Experiments on noisy-reverberant benchmarks, including WHAMR! and MC-Libri2Mix, demonstrate the effectiveness of MC-TSE.
BibTeX
@misc{ling2025mc,
  author = {Tongtao Ling and S. He and Pengjie Shen and Zhong-Qiu Wang},
  title = {{MC-LExt: Multi-Channel Target Speaker Extraction with Onset-Prompted Speaker Conditioning Mechanism}},
  year = {2025},
  eprint = {2510.15437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15437},
  howpublished = {arXiv},
}

Loading related content...

Figure