CasNet: Investigating Channel Robustness for Speech Separation

Fan-Lin Wang, Yao-Fei Cheng, Hung-Shin Lee, Yu Tsao, Hsin‐Min Wang

Abstract

Recording channel mismatch between training and testing conditions has been shown to be a serious problem for speech separation. This situation greatly reduces the separation performance, and cannot meet the requirement of daily use. In this study, inheriting the use of our previously constructed TAT-2mix corpus, we address the channel mismatch problem by proposing a channel-aware audio separation network (CasNet), a deep learning framework for end-to-end time-domain speech separation. CasNet is implemented on top of TasNet. Channel embedding (characterizing channel information in a mixture of multiple utterances) generated by Channel Encoder is introduced into the separation module by the FiLM technique. Through two training strategies, we explore two roles that channel embedding may play: 1) a real-life noise disturbance, making the model more robust, or 2) a guide, instructing the separation model to retain the desired channel information. Experimental results on TAT-2mix show that CasNet trained with both training strategies outperforms the TasNet baseline, which does not use channel embeddings.

Introduction: The Channel Mismatch Problem

Speech separation—the "cocktail party problem" of isolating individual voices from a noisy mixture—has seen massive leaps in performance thanks to deep learning. However, most models are trained and tested on "matched" data, where the recording conditions are identical.

In the real world, recording devices vary significantly, from high-end condenser microphones to tiny MEMS sensors in smartphones. When a model trained on one type of microphone is asked to separate speech recorded on another (a "channel mismatch"), performance usually plummets. This channel mismatch is often a larger hurdle for model generalization than differences in language or vocabulary.

Methodology

CasNet Architecture

CasNet is built upon the popular TasNet (Time-domain Audio Separation Network) architecture. It introduces two primary innovations to handle channel variability:

  1. Channel Encoder: This sub-network consists of a Residual Net with Squeeze-and-Excitation (SE) blocks and an attentive pooling layer. It processes a mixture of voices to extract a "channel embedding"—a vector characterizing the specific recording hardware and environment.
  2. FiLM Integration: To inject this channel information into the separation process, the model utilizes FiLM (Feature-wise Linear Modulation). This technique applies a learned linear transformation (scaling and shifting) to the internal features of the separation model based on the channel embedding.
Architecture of CasNet, integrating a Channel Encoder and FiLM into the TasNet framework.

Training Strategies

The authors explored two roles for these embeddings:

  • Guiding Strategy: The embedding acts as a reference, providing the model with the specific characteristics of the current channel.
  • Disturbing Strategy: The embedding acts as a form of data perturbation, forcing the model to become robust to varying channel interference.

Experimental Setup

The study utilized the TAT-2mix corpus, featuring six simultaneous recording channels: a close-talk mic (AT2020), an X-Y stereo mic, a lavalier mic, iOS devices, and Android phones. To test robustness, models were trained on five channels and evaluated on the unseen Android channel.

Results and Discussion

Performance was measured using SI-SNR (Scale-Invariant Signal-to-Distortion Ratio) in decibels (dB). Results indicate that explicitly modeling the recording channel helps the network navigate hardware-induced distortions more effectively than simply increasing model size.

Comparison of SI-SNR (dB) performance between CasNet and baseline models.

Key findings include:

  • Architecture vs. Size: A "Large TasNet" baseline with 19.4M parameters achieved 9.72 dB. CasNet, with fewer parameters (17.1M), outperformed it with 9.76 dB using the guiding strategy.
  • Mixtures over Utterances: Processing a mixture rather than a single-speaker clip ensures the embedding captures the microphone's "fingerprint" rather than individual voice characteristics.
  • Channel Identification Loss: This loss was found to be crucial for the "disturbing" strategy to ensure the encoder effectively distinguishes between microphones.

Conclusion

CasNet demonstrates that channel-aware modeling significantly improves speech separation robustness against microphone mismatch. Future work will focus on Channel Enhancement, aiming to not only separate voices but also improve their quality to match professional studio standards, and refining the perturbation balance in the disturbing strategy.

Summary updated

Abstract

Recording channel mismatch between training and testing conditions has been shown to be a serious problem for speech separation. This situation greatly reduces the separation performance, and cannot meet the requirement of daily use. In this study, inheriting the use of our previously constructed TAT-2mix corpus, we address the channel mismatch problem by proposing a channel-aware audio separation network (CasNet), a deep learning framework for end-to-end time-domain speech separation. CasNet is implemented on top of TasNet. Channel embedding (characterizing channel information in a mixture of multiple utterances) generated by Channel Encoder is introduced into the separation module by the FiLM technique. Through two training strategies, we explore two roles that channel embedding may play: 1) a real-life noise disturbance, making the model more robust, or 2) a guide, instructing the separation model to retain the desired channel information. Experimental results on TAT-2mix show that CasNet trained with both training strategies outperforms the TasNet baseline, which does not use channel embeddings.
BibTeX
@misc{wang2022casnet,
  author = {Fan-Lin Wang and Yao-Fei Cheng and Hung-Shin Lee and Yu Tsao and Hsin‐Min Wang},
  title = {{CasNet: Investigating Channel Robustness for Speech Separation}},
  year = {2022},
  eprint = {2210.15370},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.15370},
}
Figure