Abstract
Existing generative adversarial networks (GANs) for speech enhancement solely rely on the convolution operation, which may obscure temporal dependencies across the sequence input. To remedy this issue, we propose a self-attention layer adapted from non-local attention, coupled with the convolutional and deconvolutional layers of a speech enhancement GAN (SEGAN) using raw signal input. Further, we empirically study the effect of placing the self-attention layer at the (de)convolutional layers with varying layer indices as well as at all of them when memory allows. Our experiments show that introducing self-attention to SEGAN leads to consistent improvement across the objective evaluation metrics of enhancement performance. Furthermore, applying at different (de)convolutional layers does not significantly alter performance, suggesting that it can be conveniently applied at the highest-level (de)convolutional layer with the smallest memory overhead.
1. Introduction
Speech enhancement is the critical task of isolating a clean voice signal from noisy background environments. While deep learning has significantly advanced this field, Generative Adversarial Networks (GANs) have emerged as a powerful tool, often producing fewer audio artifacts than traditional discriminative models. However, foundational architectures like the Speech Enhancement GAN (SEGAN) rely almost exclusively on convolutional layers.
The primary limitation of convolutions is their local "receptive field," which makes it difficult to capture long-range temporal dependencies in high-resolution audio sequences. While Recurrent Neural Networks (RNNs) can model such dependencies, their sequential nature prevents efficient parallelization. To address this, we introduce the Self-Attention SEGAN (SASEGAN), which integrates a self-attention mechanism to efficiently model both local and global dependencies in raw speech signals.
2. Proposed Method: SASEGAN
The SASEGAN architecture adapts a "non-local attention" mechanism into the standard 1D convolutional and deconvolutional layers of the baseline SEGAN. Processing raw audio sequences (e.g., 16,384 samples) presents a significant computational challenge, as standard self-attention has $O(n^2)$ complexity.
To maintain memory efficiency, we implement two key modifications:
- Channel Reduction: We reduce the channel dimension of the query (Q), key (K), and value (V) matrices using $1 \times 1$ convolutions by a factor of $k=8$.
- Temporal Downsampling: We shrink the time dimension of the key and value matrices using max pooling by a factor of $p=4$.
The generator follows an encoder-decoder structure with skip connections. We experimented with inserting the self-attention layer at various depths (layers 3 through 11) to identify the optimal balance between performance gains and memory overhead.
3. Experimental Setup
We evaluated SASEGAN using a benchmark dataset derived from the Voice Bank corpus. The training set included 40 noisy conditions (10 noise types at four SNRs: 0, 5, 10, and 15 dB) across 28 speakers. Testing was conducted on 20 unseen noisy conditions using 5 noise types from the Demand database at SNRs of 2.5, 7.5, 12.5, and 17.5 dB. All audio was downsampled to 16 kHz for processing.
4. Results and Discussion
SASEGAN consistently outperformed the baseline SEGAN across all objective metrics, including PESQ (Perceptual Evaluation of Speech Quality), SSNR (Segmental SNR), and STOI (Short-Time Objective Intelligibility).
4.1 Performance Gains
Models with attention at a single layer (SASEGAN-3 through 11) showed an average PESQ gain of 0.15 (2.34 vs. 2.19) and an SSNR gain of 0.69 (8.05 vs. 7.36). The "SASEGAN-All" configuration, which applies attention at all feasible layers, achieved the highest performance (PESQ: 2.36, SSNR: 8.27), albeit with higher computational costs.
4.2 Global Context Visualization
A key finding is that the network actively leverages global context. Visualizing the attention weights demonstrates that the model learns to synthesize features by attending to distant, complementary regions of the audio sequence rather than relying solely on local neighborhoods.
4.3 Efficiency of Deep Placement
Interestingly, placing the self-attention layer at deeper layers (where the time dimension is smaller due to downsampling) did not result in a significant performance drop. This suggests that self-attention can be applied at the highest-level layers to achieve substantial gains with minimal memory overhead.
5. Conclusion
The integration of self-attention into SEGAN provides a robust method for capturing long-range dependencies in speech signals. While memory constraints prevented testing at the very earliest layers, the results indicate that deep placement is both effective and efficient. Future work will explore integrating this generic self-attention module into more advanced SEGAN variants like ISEGAN or DSEGAN to further push the state-of-the-art in speech enhancement.
Summary updated