Speech Enhancement using Separable Polling Attention and Global Layer Normalization followed with PReLU

Dengfeng Ke, Jinsong Zhang, Yanlu Xie, Yanyan Xu, Binghuai LinarXiv 2021 · OpenReview

Abstract

Single-channel speech enhancement remains a challenging task in the speech processing community. While models like PHASEN and T-GSA have achieved state-of-the-art performance on the VoiceBank+DEMAND corpus, their large parameter counts (e.g., 33M for PHASEN) limit their practical deployment. This paper introduces three key techniques to shrink the PHASEN model while simultaneously improving its performance: Separable Polling Attention (SPA), Global Layer Normalization (GLN) with PReLU activations, and architecture pruning of the PostNet. These modifications reduce the model size from 33M to 5M parameters—an 85% reduction—while achieving a PESQ score of 3.07 and a CSIG score of 4.30, outperforming the original baseline.

Introduction

In speech enhancement, there is a persistent trade-off between model size and audio quality. Top-tier models often rely on computationally heavy components like Frequency Transformation Blocks (FTBs) and Bidirectional LSTMs (BLSTMs). This research addresses the contradiction between performance and model size by redesigning the PHASEN architecture for efficiency. The goal is to create a lightweight, "phase-aware" model suitable for real-time applications without sacrificing the high-fidelity output expected from state-of-the-art systems.

System Architecture

The proposed system follows a modular design consisting of a preliminary network (PrevNet), a core processing stage using Two-Stream Blocks, and a final masking stage (PostNet). The two-stream approach allows the model to process amplitude and phase information in parallel, which is critical for high-quality speech reconstruction.

Overall architecture of the proposed speech enhancement system.

Proposed Methods

Separable Polling Attention (SPA)

The primary contribution is the Separable Polling Attention (SPA) mechanism, designed to replace the parameter-heavy FTBs. Standard attention mechanisms often struggle with the high dimensionality of frequency-time representations. SPA decomposes the attention operation into three distinct, efficient steps: a $1 \times 1$ convolution in the channel dimension, followed by convolutions in the frequency and time dimensions. This "separable" approach reduces the attention block's parameter count from 2.8 million to just 60,000.

Detail of the proposed Separable Polling Attention (SPA) mechanism.

Global Layer Normalization and PReLU

To improve convergence and handle the nuances of phase data, the authors replaced standard Batch Normalization and ReLU activations. Global Layer Normalization (GLN) was found to be more robust for speech tasks, while Parametric ReLU (PReLU) prevents the "dying ReLU" problem in the phase stream, allowing the model to learn more complex phase relationships.

PostNet Pruning

The original PHASEN model utilized a massive BLSTM in its PostNet. By replacing this with a simpler 2D Convolutional layer, the authors saved 11 million parameters. This change reflects the insight that for the final masking stage, local spatial-temporal features captured by convolutions are as effective as the long-range dependencies captured by LSTMs, but at a fraction of the cost.

Experimental Results

The model was trained and evaluated on the VoiceBank+DEMAND corpus. The results demonstrate that the 5M-parameter SPA-LN-PReLU model outperforms the 33M-parameter PHASEN baseline across all standard metrics:

  • PESQ: Improved from 2.99 to 3.07
  • CSIG: Improved from 4.21 to 4.30
  • COVL: Improved from 3.62 to 3.73

Furthermore, the model exhibited faster convergence during training, reaching higher performance levels in fewer epochs than the original architecture.

Conclusion

By applying principles of efficiency—specifically separable operations and targeted architectural pruning—this work demonstrates that state-of-the-art speech enhancement does not require massive parameter counts. The proposed model achieves an 85% reduction in size while setting new performance benchmarks, paving the way for high-quality speech enhancement on resource-constrained edge devices.

Summary updated

Abstract

Single channel speech enhancement is a challenging task in speech community. Recently, various neural networks based methods have been applied to speech enhancement. Among these models, PHASEN and T-GSA achieve state-of-the-art performances on the publicly opened VoiceBank+DEMAND corpus. Both of the models reach the COVL score of 3.62. PHASEN achieves the highest CSIG score of 4.21 while T-GSA gets the highest PESQ score of 3.06. However, both of these two models are very large. The contradiction between the model performance and the model size is hard to reconcile. In this paper, we introduce three kinds of techniques to shrink the PHASEN model and improve the performance. Firstly, seperable polling attention is proposed to replace the frequency transformation blocks in PHASEN. Secondly, global layer normalization followed with PReLU is used to replace batch normalization followed with ReLU. Finally, BLSTM in PHASEN is replaced with Conv2d operation and the phase stream is simplified. With all these modifications, the size of the PHASEN model is shrunk from 33M parameters to 5M parameters, while the performance on VoiceBank+DEMAND is improved to the CSIG score of 4.30, the PESQ score of 3.07 and the COVL score of 3.73.
BibTeX
@misc{ke2021speech,
  author = {Dengfeng Ke and Jinsong Zhang and Yanlu Xie and Yanyan Xu and Binghuai Lin},
  title = {{Speech Enhancement using Separable Polling Attention and Global Layer Normalization followed with PReLU}},
  year = {2021},
  eprint = {2105.02509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.02509},
}
Figure