ctPuLSE: Close-Talk, and Pseudo-Label Based Far-Field, Speech Enhancement

Zhong-Qiu Wang

Abstract

Deep learning models for speech enhancement often suffer from a "generalization gap" when moving from simulated training data to real-world recordings. This paper proposes ctPuLSE, a framework that leverages close-talk (CT) microphone recordings as high-quality pseudo-labels to bridge this gap. By combining simulated data with real-world recordings aligned through a learnable linear filter, ctPuLSE achieves significant improvements in speech quality and automatic speech recognition (ASR) performance on the CHiME-4 benchmark.

Introduction

The standard approach to training speech enhancement models involves supervised learning on simulated mixtures of clean speech and noise. However, these simulations often fail to capture the complexities of real-world acoustics, such as specific microphone characteristics and diverse noise types. Consequently, models that perform well on simulated test sets often see a performance drop when applied to real-world data.

The physical setup of a reverberant room showing the relationship between the target speaker, close-talk microphone, and far-field microphone array.

To address this, we propose using real-world data for training. Since ground-truth clean speech is unavailable for real recordings, we utilize the close-talk microphone—a device often present during data collection but typically ignored—as a source for "pseudo-labels."

Methods

The proposed system, ctPuLSE (Close-Talk and Pseudo-Label Based Speech Enhancement), follows a multi-step pipeline designed to refine and align these pseudo-labels for effective supervision.

Close-Talk Speech Enhancement (CTSEnet)

While close-talk microphones have a high signal-to-noise ratio, they still capture some ambient noise. We first train a model, CTSEnet, on simulated data to enhance the real-recorded close-talk mixtures, creating cleaner pseudo-labels for the subsequent far-field training.

Signal Alignment

A major challenge in using CT recordings as labels for far-field (FF) inputs is the spatial and temporal discrepancy between the two microphones. We employ a two-step alignment process:

  • Rough Synchronization: A modified GCC-PHAT algorithm aligns the signals in time.
  • Fine-Grained Alignment: During training, a learnable linear filter mathematically aligns the model's far-field estimate with the CT pseudo-label before the loss is calculated.

Co-Learning and Training Logic

The final model, ctPuLSEnet, is trained using a co-learning strategy. It simultaneously learns from simulated data (with perfect ground-truth labels) and real-world data (with pseudo-labels). This approach prevents the model from overfitting to the specific artifacts of either dataset. The underlying architecture for both models is TF-GridNet, a state-of-the-art deep neural network for spectral mapping.

The training framework for ctPuLSE, illustrating supervised training on both real mixtures (using pseudo-labels) and simulated mixtures.

Results

We evaluated ctPuLSE on the CHiME-4 dataset, which includes real-recorded mixtures in challenging environments like buses and cafes.

  • Speech Quality: On the real test set, ctPuLSE improved the DNSMOS score from 1.39 (unprocessed) to 3.29, outperforming models trained solely on simulations (3.21).
  • ASR Performance: In monaural scenarios, purely supervised models often introduce artifacts that increase the Word Error Rate (WER). ctPuLSE reduced the WER to 3.13%, compared to 4.47% for unprocessed audio. In 6-channel configurations, it achieved a state-of-the-art WER of 1.56%.

Conclusion

By treating close-talk microphones as a "goldmine" for real-world supervision, ctPuLSE effectively bridges the gap between simulation and reality. The combination of pseudo-labeling, precise alignment, and co-learning provides a robust framework for enhancing speech in complex, real-world acoustic environments.

Summary updated

BibTeX
@article{wang2025ctpulse,
  author = {Zhong-Qiu Wang},
  title = {{ctPuLSE: Close-Talk, and Pseudo-Label Based Far-Field, Speech Enhancement}},
  journal = {The Journal of the Acoustical Society of America},
  year = {2025},
  doi = {10.1121/10.0039557},
}
Figure