Abstract
Single-channel speech enhancement remains a challenging task in the speech processing community. While models like PHASEN and T-GSA have achieved state-of-the-art performance on the VoiceBank+DEMAND corpus, their large parameter counts (e.g., 33M for PHASEN) limit their practical deployment. This paper introduces three key techniques to shrink the PHASEN model while simultaneously improving its performance: Separable Polling Attention (SPA), Global Layer Normalization (GLN) with PReLU activations, and architecture pruning of the PostNet. These modifications reduce the model size from 33M to 5M parameters—an 85% reduction—while achieving a PESQ score of 3.07 and a CSIG score of 4.30, outperforming the original baseline.
Introduction
In speech enhancement, there is a persistent trade-off between model size and audio quality. Top-tier models often rely on computationally heavy components like Frequency Transformation Blocks (FTBs) and Bidirectional LSTMs (BLSTMs). This research addresses the contradiction between performance and model size by redesigning the PHASEN architecture for efficiency. The goal is to create a lightweight, "phase-aware" model suitable for real-time applications without sacrificing the high-fidelity output expected from state-of-the-art systems.
System Architecture
The proposed system follows a modular design consisting of a preliminary network (PrevNet), a core processing stage using Two-Stream Blocks, and a final masking stage (PostNet). The two-stream approach allows the model to process amplitude and phase information in parallel, which is critical for high-quality speech reconstruction.
Proposed Methods
Separable Polling Attention (SPA)
The primary contribution is the Separable Polling Attention (SPA) mechanism, designed to replace the parameter-heavy FTBs. Standard attention mechanisms often struggle with the high dimensionality of frequency-time representations. SPA decomposes the attention operation into three distinct, efficient steps: a $1 \times 1$ convolution in the channel dimension, followed by convolutions in the frequency and time dimensions. This "separable" approach reduces the attention block's parameter count from 2.8 million to just 60,000.
Global Layer Normalization and PReLU
To improve convergence and handle the nuances of phase data, the authors replaced standard Batch Normalization and ReLU activations. Global Layer Normalization (GLN) was found to be more robust for speech tasks, while Parametric ReLU (PReLU) prevents the "dying ReLU" problem in the phase stream, allowing the model to learn more complex phase relationships.
PostNet Pruning
The original PHASEN model utilized a massive BLSTM in its PostNet. By replacing this with a simpler 2D Convolutional layer, the authors saved 11 million parameters. This change reflects the insight that for the final masking stage, local spatial-temporal features captured by convolutions are as effective as the long-range dependencies captured by LSTMs, but at a fraction of the cost.
Experimental Results
The model was trained and evaluated on the VoiceBank+DEMAND corpus. The results demonstrate that the 5M-parameter SPA-LN-PReLU model outperforms the 33M-parameter PHASEN baseline across all standard metrics:
- PESQ: Improved from 2.99 to 3.07
- CSIG: Improved from 4.21 to 4.30
- COVL: Improved from 3.62 to 3.73
Furthermore, the model exhibited faster convergence during training, reaching higher performance levels in fewer epochs than the original architecture.
Conclusion
By applying principles of efficiency—specifically separable operations and targeted architectural pruning—this work demonstrates that state-of-the-art speech enhancement does not require massive parameter counts. The proposed model achieves an 85% reduction in size while setting new performance benchmarks, paving the way for high-quality speech enhancement on resource-constrained edge devices.
Summary updated