Cross-Talk Reduction

Zhong-Qiu Wang, Anurag Kumar, Shinji Watanabe

Abstract

While far-field multi-talker mixtures are recorded, each speaker can wear a close-talk microphone so that close-talk mixtures can be recorded at the same time. Although each close-talk mixture has a high signal-to-noise ratio (SNR) of the wearer, it has a very limited range of applications, as it also contains significant cross-talk speech by other speakers and is not clean enough. In this context, we propose a novel task named cross-talk reduction (CTR) which aims at reducing cross-talk speech, and a novel solution named CTRnet which is based on unsupervised or weakly-supervised neural speech separation. In unsupervised CTRnet, close-talk and far-field mixtures are stacked as input for a DNN to estimate the close-talk speech of each speaker. It is trained in an unsupervised, discriminative way such that the DNN estimate for each speaker can be linearly filtered to cancel out the speaker's cross-talk speech captured at other microphones. In weakly-supervised CTRnet, we assume the availability of each speaker's activity timestamps during training, and leverage them to improve the training of unsupervised CTRnet. Evaluation results on a simulated two-speaker CTR task and on a real-recorded conversational speech separation and recognition task show the effectiveness and potential of CTRnet.

Introduction

In multi-talker environments like meetings or dinner parties, audio is often captured using both far-field microphone arrays and individual close-talk microphones (e.g., lapels). While close-talk microphones provide a high signal-to-noise ratio (SNR) for the wearer, they are frequently contaminated by cross-talk—the voices of other nearby speakers. This "bleeding" makes the recordings unsuitable as clean references for evaluation or training.

Illustration of the cross-talk reduction (CTR) task in a reverberant room.

The proposed Cross-Talk Reduction (CTR) task aims to "de-bleed" these close-talk recordings by leveraging the multi-microphone data available in such setups. Traditional blind source separation methods often struggle with real-world acoustic complexities, creating a need for a dedicated, robust solution.

Methodology: CTRnet

The authors introduce CTRnet, a framework that treats CTR as an unsupervised blind deconvolution problem. The core architecture utilizes TF-GridNet to handle complex spectral mapping.

Unsupervised CTRnet

The model takes raw audio from all available microphones (close-talk and far-field) as input. Instead of requiring ground-truth clean labels, it is trained using a Mixture-Constraint (MC) Loss. The DNN estimates "dry" close-talk speech for each speaker, which is then processed through a Forward Convolutive Prediction (FCP) filter. This filter re-reverberates the estimates to match the physical mixtures at every microphone. The model learns by minimizing the difference between these reconstructed mixtures and the original recordings.

Overview of the CTRnet architecture with Mixture-Constraint (MC) loss and Forward Convolutive Prediction (FCP).

Weakly-Supervised CTRnet

To improve performance in real-world scenarios with sparse overlap, the authors incorporate speaker activity timestamps (VAD). This weak supervision allows the model to:

  • Mute outputs during known silent periods for a specific speaker during training.
  • Apply a Speaker-Activity (SA) Loss to penalize predictions in silent regions.

Experiments and Results

CTRnet was evaluated on two benchmarks:

  1. SMS-WSJ-FF-CT (Simulated): A two-speaker reverberant environment. CTRnet improved the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) from 14.7 dB (unprocessed) to 26.5 dB, significantly outperforming Independent Vector Analysis (IVA) baselines.
  2. CHiME-7 (Real-world): A challenging dataset with four speakers and significant noise. Performance was measured via Diarization-assigned Word Error Rate (DA-WER). Weakly-supervised CTRnet reduced the DA-WER from 27.8% (unprocessed) to 22.6%, surpassing the Guided Source Separation (GSS) baseline of 26.6%.

Discussion and Conclusion

The success of CTRnet stems from using close-talk mixtures as a strong prior, providing the model with a high-SNR "hint" that simplifies the separation task. The FCP mechanism allows the model to implicitly learn acoustic transfer functions.

However, challenges remain regarding synchronization and speaker mobility. While the model handles minor jitter, significant clock drift between devices can degrade FCP performance. Additionally, the current formulation assumes relatively stationary speakers within short processing blocks. Future work will focus on optimizing computational complexity and adapting to highly mobile environments.

Summary updated

Abstract

While far-field multi-talker mixtures are recorded, each speaker can wear a close-talk microphone so that close-talk mixtures can be recorded at the same time. Although each close-talk mixture has a high signal-to-noise ratio (SNR) of the wearer, it has a very limited range of applications, as it also contains significant cross-talk speech by other speakers and is not clean enough. In this context, we propose a novel task named \textit{cross-talk reduction} (CTR) which aims at reducing cross-talk speech, and a novel solution named CTRnet which is based on unsupervised or weakly-supervised neural speech separation. In unsupervised CTRnet, close-talk and far-field mixtures are stacked as input for a DNN to estimate the close-talk speech of each speaker. It is trained in an unsupervised, discriminative way such that the DNN estimate for each speaker can be linearly filtered to cancel out the speaker's cross-talk speech captured at other microphones. In weakly-supervised CTRnet, we assume the availability of each speaker's activity timestamps during training, and leverage them to improve the training of unsupervised CTRnet. Evaluation results on a simulated two-speaker CTR task and on a real-recorded conversational speech separation and recognition task show the effectiveness and potential of CTRnet.
BibTeX
@inproceedings{wang2024cross,
  author = {Zhong-Qiu Wang and Anurag Kumar and Shinji Watanabe},
  title = {{Cross-Talk Reduction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=NRLDnVZ9tb},
}
Figure