Cross-Talk Speech Reduction, by Separation, for Separation

Zhong-Qiu Wang, Samuele CornellarXiv 2026 · arXiv

TL;DR

This paper introduces a novel two-stage framework to train deep learning speech separation models directly on real-world conversational data, bypassing the domain mismatch issues typical of synthetically trained models. By first cleaning up cross-talk in close-talk (lapel) microphone recordings using a new method called CTRnet, the authors generate high-quality pseudo-labels to train a far-field speech separation model called PuLSS. The resulting system achieves state-of-the-art automatic speech recognition (ASR) performance on the notoriously difficult CHiME-6 dataset, outperforming standard signal-processing baselines.

What Problem Does This Paper Address?

In conversational speech recognition (the "cocktail party problem"), the goal is to isolate individual speakers from a mixture recorded by far-field microphone arrays. Currently, the dominant approach relies on supervised deep learning models trained on massive synthetic datasets. While these models perform beautifully on simulations, they often fail in real-world deployments due to a severe domain mismatch. Real environments introduce unmodeled complexities like moving speakers, microphone synchronization errors, signal clipping, and non-stationary background noise.

To train directly on real data, researchers need target labels. During data collection for conversational datasets, it is standard practice to have speakers wear close-talk microphones (like lapels) to help with manual transcription.

Fig. 1: Typical setup for collecting training data in conversational scenarios, where a microphone array is placed at far field to record far-field mixtures. For annotation purposes, a standard practice is to have each speaker wear a close-talk microphone to record close-talk mixtures so that data annotation (e.g., word transcriptions and annotating speaker-activity timestamps) can be facilitated. At deployment, only the far-field array is available for inference.

Intuitively, these close-talk recordings could serve as the "clean" target labels for training far-field separation models. However, they aren't actually clean; they contain significant cross-talk from nearby speakers and ambient noise. Because they are mixtures themselves, they cannot be used directly as supervised targets.

Methods

The authors propose a two-stage framework: Cross-Talk Reduction (CTR) followed by Far-Field Speech Separation.

Fig. 2: System overview. (a) Training Stage: CTRnet is trained in a semi- supervised manner on real-recorded pairs of close-talk and far-field mixtures to estimate close-talk speech (see Section IV-D). The estimate is then used as pseudo-labels for training PuLSS in a supervised fashion on real-recorded far- field mixtures (see Section V-D). In PuLSS, oracle speaker-activity timestamps are used in input features to resolve the speaker-permutation problem during training. (b) Inference Stage: the PuLSS model separates far-field mixtures and an ASR model transcribes the separated speech. At inference time, either oracle speaker-activity timestamps or estimated ones by an external speaker diarization system can be used (see the dashed arrow).
  1. CTRnet (Cross-Talk Reduction Network): This model isolates the wearer's speech from their close-talk microphone mixture. It formulates CTR as a blind deconvolution problem. Instead of relying on clean targets, CTRnet is trained using a Mixture-Constraint (MC) loss. This loss enforces a physical acoustic constraint: the estimated clean close-talk signals, when linearly filtered using Forward Convolutive Prediction (FCP), must be able to mathematically reconstruct the observed close-talk and far-field mixtures. The authors extend this with weak supervision (using speaker-activity timestamps to mute silent frames) and semi-supervised learning (combining real and simulated data).
  2. PuLSS (Pseudo-Label based far-field Speech Separation): Once CTRnet is trained, its clean speech estimates act as high-quality pseudo-labels. PuLSS uses FCP to project these close-talk estimates to a reference far-field microphone, creating a pseudo-target speaker image. A supervised far-field separation model (built on the TF-GridNet architecture) is then trained to predict these pseudo-targets from the real far-field mixtures. To avoid the classic source permutation ambiguity over long audio files, the model is explicitly conditioned on speaker-activity timestamps.

Datasets

  • CHiME-6: The primary dataset used for training and evaluation. It features real-recorded, unconstrained dinner party conversations across multiple rooms, characterized by severe cross-talk, moving speakers, and realistic domestic noise.
  • Simulation Sources: For the semi-supervised training components, synthetic mixtures were generated using clean speech from LibriSpeech and EARS, noise from FSD50K, and room impulse responses simulated via the Pyroomacoustics toolkit, supplemented by air-conditioning noise from the REVERB dataset.

Key Insights and Ideas

  • Separation for Separation: The core "aha moment" is using a neural network to clean up the supervision signal itself so that a second neural network can successfully learn the primary task.
  • Physics-Guided Unsupervised Learning: Rather than letting the unsupervised network learn arbitrarily, the Mixture-Constraint loss anchors the network's predictions to the physical realities of acoustic wave propagation and linear filtering.
  • Timestamp Conditioning over Stitching: Instead of using Permutation Invariant Training (PIT) and complex block-stitching algorithms for long-form audio, simply feeding speaker-activity timestamps as input features completely resolves speaker identity across time blocks.

Key Results

The system was evaluated using concatenated minimum permutation word error rate (cpWER) for oracle diarization and time-constrained cpWER (tcpWER) for estimated diarization.

  • Close-Talk Cleaning: CTRnet successfully reduced the cpWER of raw close-talk mixtures from 29.4% to 15.0%, demonstrating its ability to effectively remove cross-talk.
  • Far-Field Separation (Oracle Diarization): PuLSS achieved a cpWER of 19.5% on far-field mixtures, significantly outperforming the de-facto signal processing standard, Guided Source Separation (GSS), which scored 29.7%. It also surpassed all previous submissions to the CHiME-7 and CHiME-8 challenges.
  • Robustness to Diarization Errors: When evaluated using real, imperfect diarization systems (such as the USTC diarizer), PuLSS maintained its edge, achieving 28.5% tcpWER compared to GSS's 33.5%.

Limitations and Future Work

  • Non-Verbal Sounds: CTRnet tends to preserve non-verbal wearer sounds (breathing, chewing, laughing). Because these sounds are often too weak to be captured by far-field microphones, they introduce inconsistencies during PuLSS training.
  • Fixed Speaker Count: The current framework assumes a fixed maximum number of speakers ($C=4$), though the authors note it can be adapted to open-speaker counts using existing separation algorithms.
  • End-to-End Integration: PuLSS and the downstream ASR model are currently trained separately. Jointly fine-tuning the separation frontend with the ASR backend remains an open avenue for exploration.

Summary updated

Abstract

In conversational speech separation and recognition tasks, close-talk microphones are typically attached to each speaker during training data collection to capture near-field, close-talk mixture signals, in addition to using far-field microphones to record far-field mixture signals. Each such close-talk mixture exhibits a reasonably high energy level for the wearer and could intuitively serve as weak supervision for training far-field speech separation models directly on real-recorded far-field signals. However, they are not sufficiently clean for this purpose, as they often contain strong cross-talk speech from other speakers in addition to background noise. To address this, we propose cross-talk reduction (CTR), a task aiming to isolate the wearer's speech from each close-talk mixture, and a novel method called CTRnet, which can be trained directly on real-recorded pairs of close-talk and far-field mixtures to accomplish CTR. Building on CTRnet, we further propose pseudo-label based far-field speech separation (PuLSS), which uses CTRnet's estimated clean speech as pseudo-labels to train models for separating far-field mixtures. A key advantage of the proposed framework is that both CTRnet and PuLSS can be trained on real-recorded data from the target domain, addressing the generalization gap commonly observed when models are trained exclusively on simulated data. On the CHiME-6 dataset, our framework achieves state-of-the-art ASR performance under both oracle and estimated speaker diarization, surpassing all CHiME-{7,8} challenge submissions. To our knowledge, it is the first neural speech separation method that substantially outperforms guided source separation on real conversational "speech-in-the-wild" data.
BibTeX
@misc{wang2026cross,
  author = {Zhong-Qiu Wang and Samuele Cornell},
  title = {{Cross-Talk Speech Reduction, by Separation, for Separation}},
  year = {2026},
  eprint = {2605.19695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.19695},
}

Loading related content...

Figure