TL;DR
This paper introduces a novel two-stage framework to train deep learning speech separation models directly on real-world conversational data, bypassing the domain mismatch issues typical of synthetically trained models. By first cleaning up cross-talk in close-talk (lapel) microphone recordings using a new method called CTRnet, the authors generate high-quality pseudo-labels to train a far-field speech separation model called PuLSS. The resulting system achieves state-of-the-art automatic speech recognition (ASR) performance on the notoriously difficult CHiME-6 dataset, outperforming standard signal-processing baselines.
What Problem Does This Paper Address?
In conversational speech recognition (the "cocktail party problem"), the goal is to isolate individual speakers from a mixture recorded by far-field microphone arrays. Currently, the dominant approach relies on supervised deep learning models trained on massive synthetic datasets. While these models perform beautifully on simulations, they often fail in real-world deployments due to a severe domain mismatch. Real environments introduce unmodeled complexities like moving speakers, microphone synchronization errors, signal clipping, and non-stationary background noise.
To train directly on real data, researchers need target labels. During data collection for conversational datasets, it is standard practice to have speakers wear close-talk microphones (like lapels) to help with manual transcription.
Intuitively, these close-talk recordings could serve as the "clean" target labels for training far-field separation models. However, they aren't actually clean; they contain significant cross-talk from nearby speakers and ambient noise. Because they are mixtures themselves, they cannot be used directly as supervised targets.
Methods
The authors propose a two-stage framework: Cross-Talk Reduction (CTR) followed by Far-Field Speech Separation.
- CTRnet (Cross-Talk Reduction Network): This model isolates the wearer's speech from their close-talk microphone mixture. It formulates CTR as a blind deconvolution problem. Instead of relying on clean targets, CTRnet is trained using a Mixture-Constraint (MC) loss. This loss enforces a physical acoustic constraint: the estimated clean close-talk signals, when linearly filtered using Forward Convolutive Prediction (FCP), must be able to mathematically reconstruct the observed close-talk and far-field mixtures. The authors extend this with weak supervision (using speaker-activity timestamps to mute silent frames) and semi-supervised learning (combining real and simulated data).
- PuLSS (Pseudo-Label based far-field Speech Separation): Once CTRnet is trained, its clean speech estimates act as high-quality pseudo-labels. PuLSS uses FCP to project these close-talk estimates to a reference far-field microphone, creating a pseudo-target speaker image. A supervised far-field separation model (built on the TF-GridNet architecture) is then trained to predict these pseudo-targets from the real far-field mixtures. To avoid the classic source permutation ambiguity over long audio files, the model is explicitly conditioned on speaker-activity timestamps.
Datasets
- CHiME-6: The primary dataset used for training and evaluation. It features real-recorded, unconstrained dinner party conversations across multiple rooms, characterized by severe cross-talk, moving speakers, and realistic domestic noise.
- Simulation Sources: For the semi-supervised training components, synthetic mixtures were generated using clean speech from LibriSpeech and EARS, noise from FSD50K, and room impulse responses simulated via the Pyroomacoustics toolkit, supplemented by air-conditioning noise from the REVERB dataset.
Key Insights and Ideas
- Separation for Separation: The core "aha moment" is using a neural network to clean up the supervision signal itself so that a second neural network can successfully learn the primary task.
- Physics-Guided Unsupervised Learning: Rather than letting the unsupervised network learn arbitrarily, the Mixture-Constraint loss anchors the network's predictions to the physical realities of acoustic wave propagation and linear filtering.
- Timestamp Conditioning over Stitching: Instead of using Permutation Invariant Training (PIT) and complex block-stitching algorithms for long-form audio, simply feeding speaker-activity timestamps as input features completely resolves speaker identity across time blocks.
Key Results
The system was evaluated using concatenated minimum permutation word error rate (cpWER) for oracle diarization and time-constrained cpWER (tcpWER) for estimated diarization.
- Close-Talk Cleaning: CTRnet successfully reduced the cpWER of raw close-talk mixtures from 29.4% to 15.0%, demonstrating its ability to effectively remove cross-talk.
- Far-Field Separation (Oracle Diarization): PuLSS achieved a cpWER of 19.5% on far-field mixtures, significantly outperforming the de-facto signal processing standard, Guided Source Separation (GSS), which scored 29.7%. It also surpassed all previous submissions to the CHiME-7 and CHiME-8 challenges.
- Robustness to Diarization Errors: When evaluated using real, imperfect diarization systems (such as the USTC diarizer), PuLSS maintained its edge, achieving 28.5% tcpWER compared to GSS's 33.5%.
Limitations and Future Work
- Non-Verbal Sounds: CTRnet tends to preserve non-verbal wearer sounds (breathing, chewing, laughing). Because these sounds are often too weak to be captured by far-field microphones, they introduce inconsistencies during PuLSS training.
- Fixed Speaker Count: The current framework assumes a fixed maximum number of speakers ($C=4$), though the authors note it can be adapted to open-speaker counts using existing separation algorithms.
- End-to-End Integration: PuLSS and the downstream ASR model are currently trained separately. Jointly fine-tuning the separation frontend with the ASR backend remains an open avenue for exploration.
Summary updated