TL;DR
The authors introduce Close-to-Distant microphone Projection (C2D projection), a novel method to generate high-quality training targets for real-world speech enhancement neural networks. By mathematically aligning and denoising close-microphone recordings (like wearable mics) to match the perspective of distant microphone arrays, C2D bypasses the need for flawed acoustic simulations. When tested on challenging dinner-party conversational datasets, an NN trained with C2D targets significantly outperformed state-of-the-art signal processing baselines.
What Problem Does This Paper Address?
Enhancing distant speech—such as capturing a conversation across a room via a smart speaker or microphone array—is notoriously difficult. The audio is typically plagued by reverberation, background noise, and overlapping speech from multiple talkers.
While deep neural networks (NNs) excel at cleaning up audio, they are data-hungry and require paired examples of "noisy/distorted" audio and its perfectly "clean" ground-truth counterpart. Because real-world clean references are nearly impossible to record simultaneously in the wild, researchers usually rely on simulators (like the image method) to synthetically add noise and reverb to clean speech. However, this creates a massive simulation-to-real gap: models trained on synthetic data often fail when deployed in complex, real-world acoustic environments.
Real-world datasets like CHiME6 circumvent this by having speakers wear close microphones (CMs) while recording the room with distant microphone arrays (DMs). But you can't just use the CM audio as a direct training target. It isn't aligned in phase or power with the distant array, and it often suffers from "cross-talk" (bleeding audio from nearby speakers). This paper solves the target-generation dilemma by mathematically projecting CM signals into the DM domain.
Methods
The core innovation is C2D projection, which estimates an optimal projection matrix to transform close-microphone inputs into clean reference signals perfectly aligned with the distant-microphone recordings.
The processing pipeline operates as follows:
- Mask Estimation: The method applies Guided Source Separation (GSS) to the close-microphone signals to estimate time-frequency masks, identifying exactly when and where the target speaker is talking.
- Covariance Estimation: These masks help calculate cross-domain spatial covariance matrices between the CM and DM domains.
- PMWF Variant: The projection matrix is derived as a variant of the Parametric Multichannel Wiener Filter (PMWF). This formulation allows a tunable parameter ($\mu$) to balance the tradeoff between reducing projection error and suppressing unwanted noise/cross-talk.
- NN Training: The generated C2D targets serve as the clean ground truth. The neural network (specifically a Deterministic Recursive Enhancement network architecture) is trained using both the raw DM signal and a GSS-enhanced signal as an auxiliary input to help it isolate the target speaker.
Datasets
The approach was validated using the CHiME ecosystem:
- CHiME6: Used for training and matched evaluation. It features real dinner party conversations recorded via 8 wearable CMs and 24 DMs across 6 arrays.
- CHiME8 Evaluation Set: Used to test generalization under mismatched conditions. This includes diverse recording scenarios across four distinct datasets: CHiME6, Dipco, Mixer6, and Notsofar1.
Key Insights and Ideas
- Simulations Aren't Enough: The paper reinforces that synthetic data limits speech enhancement accuracy in complex settings. Generating targets directly from real-world parallel data is a superior alternative.
- Alignment is Mandatory: Simply using clean audio as a target fails if it doesn't match the spatial and temporal perspective of the microphone array. As shown in the results, training an NN directly on raw CM signals yields terrible performance.
- The Power of Hybrid Systems: Instead of throwing away traditional signal processing, C2D uses it (via PMWF and GSS) to bootstrap the neural network, providing it with cleaner targets and helpful auxiliary inputs.
Key Results
The performance was primarily measured using the Time-Constrained minimum Permutation Word Error Rate (tcpWER)—a strict metric accounting for word errors, speaker identification, and timing—and DNSMOS for perceptual speech quality.
Matched Conditions (CHiME6)
When evaluated under oracle diarization (knowing exactly who spoke when), C2D targets led to clear improvements:
- Without post-filtering: C2D achieved a tcpWER of 26.54% on the evaluation set, outperforming the GSS baseline (27.37%) and an alternative All-to-Distant (A2D) projection method (27.11%).
- With post-filtering: C2D achieved the best overall tcpWER of 25.05% (a 4.61% relative error reduction over GSS).
- Baseline Failure: Training an NN directly on CM audio without any projection resulted in a disastrous tcpWER of 63.15%, proving the necessity of the C2D projection.
Mismatched Conditions (CHiME8)
To simulate a brutal real-world test, the authors evaluated the CHiME6-trained model on the CHiME8 set using estimated (imperfect) diarization labels across different room geometries. Despite the massive mismatch, C2D maintained a slight edge over GSS in three out of four scenarios (CHiME6, Dipco, and Notsofar1).
Limitations and Future Work
While robust, the method hit a snag on the Mixer6 dataset, where it performed worse than the GSS baseline. The authors attribute this to Mixer6 having a drastically different distant-microphone configuration than the CHiME6 data the NN was trained on.
Additionally, the current framework still relies on the output of a GSS-based PMWF as an auxiliary input during inference. Future work will focus on making the neural network entirely independent of GSS and improving its generalization across radically different microphone array geometries.
Summary updated