Self-Attention Generative Adversarial Network for Speech Enhancement

Huy Phan, Huy Le Nguyen, Oliver Y. Chén, Philipp Koch, Ngoc Q. K. Duong, Ian McLoughlin, Alfred Mertins

TL;DR

Generative Adversarial Networks (GANs) for speech enhancement typically rely on convolutional layers, which struggle to capture long-range temporal dependencies in audio sequences. This paper introduces a self-attention mechanism to the Speech Enhancement GAN (SEGAN) architecture, creating SASEGAN. The addition of self-attention consistently improves objective speech quality metrics while requiring minimal memory overhead when applied strategically to deeper network layers.

What Problem Does This Paper Address?

Speech enhancement—the process of isolating a clean voice signal from noisy background audio—has increasingly shifted toward deep learning. Generative Adversarial Networks (GANs) show great promise here, often producing fewer audio artifacts than discriminative models.

However, existing GAN architectures for this task (like the foundational SEGAN) rely almost exclusively on convolutional layers. Because convolutions operate on local "receptive fields," they struggle to capture long-range temporal dependencies across a long audio sequence. While Recurrent Neural Networks (RNNs) can model these temporal links, they process data sequentially and cannot be easily parallelized, making them computationally expensive for long sequences. The authors tackle the challenge of efficiently modeling both local and long-range dependencies in raw speech signals without bogging down the network's computational speed or memory.

Methods

The authors propose the Self-Attention SEGAN (SASEGAN). They adapt a "non-local attention" mechanism and couple it with the standard 1D convolutional and deconvolutional layers of a baseline SEGAN.

Because raw audio sequences are massive (e.g., 16,384 samples for about one second of audio), standard self-attention would blow up GPU memory due to its $O(n^2)$ complexity. To fix this, the authors introduce specific memory-efficiency tricks:

  • They reduce the channel dimension of the query, key, and value matrices using $1 \times 1$ convolutions (by a factor of $k=8$).
  • They shrink the time dimension of the key and value matrices using max pooling (by a factor of $p=4$).

The generator uses an encoder-decoder architecture. The self-attention layer is inserted into specific (de)convolutional layers of both the generator and the discriminator. The authors experiment with placing this layer at different depths in the network (layers 3 through 11) to find the optimal balance of performance and memory cost.

Datasets

The experiments use a standard benchmark dataset derived from the Voice Bank corpus (30 speakers).

  • Training: 40 noisy conditions created by mixing 10 noise types at four Signal-to-Noise Ratios (SNRs: 15, 10, 5, and 0 dB) using 28 speakers.
  • Testing: 20 unseen noisy conditions using 5 noise types from the Demand database at different SNRs (17.5, 12.5, 7.5, and 2.5 dB) across the remaining 2 speakers.
  • All audio was downsampled to 16 kHz.

Key Insights and Ideas

  • Global Context Matters: Visualizing the attention weights reveals that the network actively leverages complementary features from distant parts of the audio sequence, rather than just looking at the immediate local neighborhood. For instance, the model learns to pay attention to distant speech regions to help synthesize current speech features, and looks at distant background regions when synthesizing background noise.
  • Deep Placement is Highly Efficient: The authors found no significant performance drop when placing the self-attention layer at higher-level (deeper) layers of the network compared to lower-level ones. Because deeper layers have exponentially smaller time dimensions (due to successive downsampling in the encoder), placing the attention mechanism there incurs a remarkably small memory footprint while still delivering the full performance boost.

Key Results

SASEGAN consistently outperformed the baseline SEGAN across all five objective signal-quality metrics (PESQ, CSIG, CBAK, COVL, SSNR) and the STOI speech intelligibility metric.

  • Consistent Gains: Averaging across models with attention placed at single layers (SASEGAN-3 through 11), the absolute gains over the baseline were 0.15 for PESQ (2.34 vs 2.19), 0.69 for SSNR (8.05 vs 7.36), and 0.2% for STOI.
  • Maximum Performance: The best overall performance came from "SASEGAN-All" (attention applied at all feasible layers simultaneously), which achieved a PESQ of 2.36 and an SSNR of 8.27. However, this comes at the cost of increased computation time and memory.

Limitations and Future Work

  • Limitations: Due to strict GPU memory constraints, the authors were unable to test placing the self-attention layer at the very earliest layers (1st and 2nd), where the feature map time dimensions are massive (8,192 and 4,096 respectively).
  • Future Work: The proposed self-attention layer is highly generic. The authors suggest that instead of just comparing it to the baseline SEGAN, future work should integrate this attention module into more recent, state-of-the-art SEGAN variants (like ISEGAN or DSEGAN) to push their performance even further.

Summary updated

Abstract

Existing generative adversarial networks (GANs) for speech enhancement solely rely on the convolution operation, which may obscure temporal dependencies across the sequence input. To remedy this issue, we propose a self-attention layer adapted from non-local attention, coupled with the convolutional and deconvolutional layers of a speech enhancement GAN (SEGAN) using raw signal input. Further, we empirically study the effect of placing the self-attention layer at the (de)convolutional layers with varying layer indices as well as at all of them when memory allows. Our experiments show that introducing self-attention to SEGAN leads to consistent improvement across the objective evaluation metrics of enhancement performance. Furthermore, applying at different (de)convolutional layers does not significantly alter performance, suggesting that it can be conveniently applied at the highest-level (de)convolutional layer with the smallest memory overhead 1 .
BibTeX
@inproceedings{phan2021self,
  author = {Huy Phan and Huy Le Nguyen and Oliver Y. Chén and Philipp Koch and Ngoc Q. K. Duong and Ian McLoughlin and Alfred Mertins},
  title = {{Self-Attention Generative Adversarial Network for Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=CvZlJRkvHp},
}

Loading related content...

Figure