Abstract: While far-field multi-talker mixtures are recorded, each speaker can wear a close-talk microphone so that close-talk mixtures can be recorded at the same time. Although each close-talk mixture has a high signal-to-noise ratio (SNR) of the wearer, it has a very limited range of applications, as it also contains significant cross-talk speech by other speakers and is not clean enough. In this context, we propose a novel task named cross-talk reduction (CTR) which aims at reducing cross-talk speech, and a novel solution named CTRnet which is based on unsupervised or weakly-supervised neural speech separation. In unsupervised CTRnet, close-talk and far-field mixtures are stacked as input for a DNN to estimate the close-talk speech of each speaker. It is trained in an unsupervised, discriminative way such that the DNN estimate for each speaker can be linearly filtered to cancel out the speaker's cross-talk speech captured at other microphones. In weakly-supervised CTRnet, we assume the availability of each speaker's activity timestamps during training, and leverage them to improve the training of unsupervised CTRnet. Evaluation results on a simulated two-speaker CTR task and on a real-recorded conversational speech separation and recognition task show the effectiveness and potential of CTRnet.
Introduction
In audio engineering and speech processing, close-talk microphones—such as lapel mics or headsets—are often considered the gold standard for high-quality audio. However, in multi-talker environments like meetings or social gatherings, these microphones are rarely "clean." They frequently suffer from cross-talk, where the voices of nearby speakers are captured alongside the wearer's voice, albeit at a lower volume.
This "leakage" makes close-talk recordings unsuitable as clean ground truth for training AI models or for high-accuracy transcription. Traditional supervised learning approaches struggle here because perfectly clean reference recordings are rarely available for real-world acoustic environments. To address this, we introduce the task of Cross-Talk Reduction (CTR) and propose CTRnet, a framework designed to remove interfering voices using only the "dirty" recordings available.
The CTRnet Architecture
CTRnet treats cross-talk reduction as an unsupervised blind deconvolution problem. The system utilizes a Deep Neural Network (DNN) that processes all available audio streams—both close-talk mixtures and far-field microphone arrays—to estimate the "dry" (clean) speech for each speaker.
The framework relies on three core components:
- Mixture-Constraint (MC) Loss: The model operates on the principle that the total sound at any microphone is the sum of the target speaker's speech and the reverberated versions of all other speakers' voices.
- Forward Convolutive Prediction (FCP): CTRnet learns linear filters that model how sound propagates from one speaker to another speaker's microphone. The DNN estimates clean speech, which is then "re-reverberated" by FCP filters to match the observed cross-talk.
- Weak Supervision: To prevent the model from incorrectly assigning speech to silent intervals, the authors incorporate speaker activity timestamps (Voice Activity Detection) during training. This "muting" trick forces the DNN output to zero when a speaker is known to be silent, significantly improving separation performance.
Experimental Evaluation
The researchers evaluated CTRnet on two distinct datasets:
- SMS-WSJ-FF-CT: A simulated dataset based on the Wall Street Journal corpus, featuring 2-speaker mixtures with simulated close-talk and far-field microphone configurations.
- CHiME-7: A challenging real-world dataset of 4-person dinner parties, characterized by synchronization errors, environmental noise, and natural conversational dynamics.
Key Results
On the simulated SMS-WSJ task, CTRnet improved the Signal-to-Distortion Ratio (SI-SDR) from an unprocessed baseline of 14.7 dB to 26.5 dB.
In the real-world CHiME-7 evaluation, performance was measured by Word Error Rate (WER) using a pre-trained speech recognizer. While unprocessed mixtures yielded a 27.8% WER and the industry-standard Guided Source Separation (GSS) achieved 26.6% WER, the weakly-supervised CTRnet reached 22.6% WER. This represents a substantial improvement over existing signal processing methods.
Discussion and Conclusion
CTRnet effectively leverages the physical properties of close-talk microphones—specifically that the wearer's voice is the dominant signal—as a prior for the DNN. By combining physics-informed linear filtering (FCP) with the flexible estimation capabilities of deep learning, the model overcomes the "over-separation" issues common in unsupervised tasks.
While the current iteration of CTRnet shows great promise, future work will need to address challenges such as severe microphone clock drift, non-stationary background noise, and the dynamic acoustics of moving speakers. Nonetheless, by transforming "leaky" recordings into a viable source of training data, CTRnet provides a powerful tool for automated transcription and the development of next-generation speech AI.
Summary updated