FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement

Bin Ma, Shengkui Zhao, Bin Ma 0001, Karn N. Watcharasupat, Woon-Seng GanICASSP 2022 · DOI

TL;DR

This paper introduces FRCRN, a novel deep learning architecture for monaural speech enhancement that significantly improves feature representation by incorporating "frequency recurrence." This new approach captures long-range correlations across frequency bands, leading to state-of-the-art performance on benchmark datasets and a strong showing in a major speech enhancement challenge.

What Problem Does This Paper Address?

Monaural speech enhancement, the task of separating a target speech signal from background noise using only a single microphone, is a long-standing challenge. While deep learning, particularly convolutional recurrent networks (CRNs), has shown great promise, existing CRNs often struggle to capture long-range dependencies across the frequency spectrum. This limitation stems from the restricted receptive fields of standard convolutional layers, which primarily focus on local temporal-spectral patterns. The paper aims to overcome this by developing a method that explicitly models these crucial frequency correlations.

Methods

The core innovation is the Convolutional Recurrent Encoder-Decoder (CRED) structure. Unlike traditional CRNs that rely solely on convolutions for feature extraction, CRED introduces frequency recurrence after each convolutional layer. This is achieved by treating feature maps along the frequency axis as sequences and processing them with a Feedforward Sequential Memory Network (FSMN). This FSMN efficiently captures long-range frequency correlations.

The overall framework, Frequency Recurrent CRN (FRCRN), combines this CRED structure with a time-recurrent module (also using CFSMN) to model temporal dependencies. The model operates in the complex-valued domain, predicting a complex Ideal Ratio Mask (cIRM) to enhance the noisy spectrogram. Optimization employs a joint loss function that includes both a time-domain SI-SNR loss and a frequency-domain mask loss.

Key components include:

  • CRED: Encoder-decoder structure with stacked "CR blocks" (complex Conv2d, complex BatchNorm, LeakyReLU, and CFSMN).
  • Frequency Recurrence: Applied within CR blocks to capture long-range frequency dependencies.
  • CFSMN: Used for both frequency recurrence (within CRED) and temporal recurrence (in the main recurrent module), chosen for its efficiency compared to LSTMs.
  • Complex-valued operations: To preserve phase information crucial for speech quality.
  • Joint Loss Function: Combines SI-SNR and cIRM mask loss for comprehensive optimization.

Datasets

  • WSJO corpus: Used for wideband speech enhancement evaluation.
  • DNS-2020 dataset: A benchmark for wideband speech enhancement.
  • VoiceBank+Demand dataset: Another benchmark for wideband speech enhancement.
  • DNS-2022 dataset: Used for fullband speech enhancement evaluation.

The authors generated training and development sets by combining clean speech with noise and reverberation at various Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Frequency recurrence is key: The paper strongly suggests that explicitly modeling long-range frequency correlations is a significant factor in improving speech enhancement performance, complementing existing temporal modeling.
  • CRED architecture's effectiveness: The proposed CRED structure, by integrating frequency recurrence, demonstrably boosts feature representation beyond what standard convolutional encoder-decoders can achieve.
  • FSMN's efficiency: Using FSMN for both frequency and temporal recurrence offers a competitive alternative to LSTMs, requiring fewer parameters while maintaining high performance.
  • Complex-valued domain is beneficial: Operating in the complex domain and predicting cIRM is crucial for preserving phase information, leading to better perceptual quality.
  • Joint loss function aids optimization: Combining time-domain and mask-based losses helps the model learn both signal reconstruction and accurate mask estimation.

Key Results

  • State-of-the-art performance: FRCRN achieved state-of-the-art (SOTA) results on wideband benchmark datasets (DNS-2020 and VoiceBank+Demand) across multiple objective metrics like PESQ, STOI, and SI-SNR.
  • Strong competition performance: FRCRN secured 2nd place in the real-time fullband track of the ICASSP 2022 Deep Noise Suppression (DNS) challenge, measured by Mean Opinion Score (MOS) and Word Accuracy (WAcc).
  • Ablation studies confirm contributions: Ablation studies showed that all components of FRCRN (CRED, frequency recurrence, attention block, recurrent module) contribute to its performance, with frequency recurrence in CRED being particularly impactful.
  • FRCRN-Lite offers efficiency: A lighter version (FRCRN-Lite) with fewer parameters still provided competitive performance, indicating a good trade-off between complexity and quality.

Limitations and Future Work

  • Computational Cost: While FSMN is more efficient than LSTM, the full FRCRN model still has a significant number of parameters (6.9M) and MACs, especially for real-time applications. The FRCRN-Lite version addresses this to some extent.
  • Asymmetrical Padding: The paper mentions using asymmetrical padding for causal convolutions, which is a standard technique but might introduce minor artifacts or complexities in certain scenarios.
  • Future Work: The authors' success suggests further exploration of frequency recurrence in other audio processing tasks and potentially more efficient implementations for ultra-low latency requirements. The paper also hints at the benefits of the attention block, which could be further investigated.

Summary updated

BibTeX
@inproceedings{ma2022frcrn,
  author = {Bin Ma and Shengkui Zhao and Bin Ma and Karn N. Watcharasupat and Woon-Seng Gan},
  title = {{FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747578},
}
Figure