Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement

Xinmeng Xu, Weiping Tu, Yuhong Yang

Abstract

Speech enhancement (SE) requires balancing the capture of global acoustic context with the preservation of local harmonic structures. While non-local attention mechanisms provide global reach, they are often computationally expensive and prone to noise interference. This paper presents the Selector-Enhancer, a novel framework that dynamically selects between local and non-local attention operations at the time-frequency (T-F) unit level. Using a reinforcement learning-based "feature-filter," the model intelligently routes signals to the most effective processing path. Experimental results on VoiceBank+DEMAND and AVSpeech+AudioSet benchmarks demonstrate that the Selector-Enhancer achieves state-of-the-art denoising quality (PESQ 3.37) with significantly lower computational overhead than existing global attention models.

1. Introduction

Modern speech enhancement (SE) systems often struggle with the trade-off between local and non-local processing. Local attention is efficient and robust for capturing harmonic structures but lacks a global view of the speaker's identity or long-term temporal patterns. Conversely, non-local attention captures global similarities but suffers from $O(T^2)$ complexity and can be misled by non-stationary noise.

The Selector-Enhancer addresses this by treating attention selection as a dynamic routing problem. Instead of applying a fixed mechanism across all audio segments, the model evaluates the noise and speech characteristics of specific T-F regions to decide the optimal processing strategy.

2. The Selector-Enhancer Framework

The architecture consists of two primary components: the Enhancer backbone and the Selector module.

2.1 Dual-Attention CNN (The Enhancer)

The Enhancer is a convolutional neural network (CNN) composed of $N$ dynamic blocks. Each block features a shared feature extraction path and two parallel attention branches:

  • Local Attention Path: Focuses on immediate T-F neighbors.
  • Non-local Attention Path: Computes global correlations across the entire spectrogram.

2.2 Feature-Filter (The Selector)

The Feature-Filter acts as a controller. It processes a low-resolution representation of the input features to generate a policy mask. This mask dictates which attention path each T-F unit should follow. By sparsifying the use of non-local attention, the model maintains high efficiency without sacrificing performance.

3. Reinforcement Learning for Dynamic Selection

Since the selection process involves discrete choices, the authors utilize the REINFORCE algorithm for training. A key innovation is the difficulty-regulated reward system. This reward function balances three objectives:

  1. Reconstruction Accuracy: Minimizing L2 loss between enhanced and clean speech.
  2. Computational Efficiency: Penalizing excessive use of the non-local path.
  3. Task Difficulty: Adjusting rewards based on the inherent noise level of the segment, ensuring the model is not unfairly penalized in highly corrupted environments.

4. Experimental Evaluation

The model was evaluated on the VoiceBank + DEMAND and AVSpeech + AudioSet datasets. It consistently outperformed recent baselines, including CRN, CNN-NL, and SA-TCN.

Comparison of spectrogram reconstruction across different enhancement methods. (a) Noisy input, (b) Clean reference, (c-e) Baseline methods, (f) Proposed Selector-Enhancer.

As shown in the figure above, the Selector-Enhancer (f) recovers harmonic structures more accurately than baseline methods (c-e), particularly in the regions highlighted by the dashed boxes. The model achieves a PESQ score of 3.37 and a STOI of 94.36% while maintaining a Real-Time Factor (RTF) of 0.63.

5. Conclusion

The Selector-Enhancer demonstrates that dynamic, context-aware routing is a powerful paradigm for speech enhancement. By selectively applying non-local attention only where it provides clear benefits, the model achieves superior denoising quality and efficiency. Future work will explore extending this dynamic selection logic to other architectural components, such as adaptive kernel sizes.

Summary updated

Abstract

Attention mechanisms, such as local and non-local attention, play a fundamental role in recent deep learning based speech enhancement (SE) systems. However, a natural speech contains many fast-changing and relatively briefly acoustic events, therefore, capturing the most informative speech features by indiscriminately using local and non-local attention is challenged. We observe that the noise type and speech feature vary within a sequence of speech and the local and non-local can respectively process different types of corrupted speech regions. To leverage this, we propose Selector-Enhancer, a dual-attention based convolution neural network (CNN) with a feature-filter that can dynamically select regions from low-resolution speech features and feed them to local or non-local attention operations. In particular, the proposed feature-filter is trained by using reinforcement learning (RL) with a developed difficulty-regulated reward that related to network performance, model complexity and “the difficulty of the SE task”. The results show that our method achieves comparable or superior performance to existing approaches. In particular, Selector-Enhancer is effective for real-world denoising, where the number and types of noise are varies on a single noisy mixture.
BibTeX
@inproceedings{xu2023selector,
  author = {Xinmeng Xu and Weiping Tu and Yuhong Yang},
  title = {{Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26622},
}
Figure