Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement

Xinmeng Xu, Weiping Tu, Yuhong Yang

Abstract

Speech enhancement (SE) requires balancing the capture of local harmonic structures with long-range non-local dependencies. Traditional approaches often apply fixed attention mechanisms, leading to computational inefficiency or noise-speech correlation errors. We present the Selector-Enhancer, a dual-attention framework that dynamically routes spectrogram regions to either local or non-local attention paths using a reinforcement learning-based feature-filter. Our approach achieves state-of-the-art performance on VoiceBank+DEMAND and large-scale AVSpeech datasets while significantly reducing computational overhead.

Introduction

In speech enhancement, the goal is to extract clean speech from noisy environments. Speech signals are inherently multi-scale, containing brief acoustic events (local) and long-term linguistic or environmental traits (non-local). While Convolutional Neural Networks (CNNs) excel at local pattern recognition, they lack the receptive field for global context. Conversely, Transformers and non-local attention mechanisms capture global dependencies but suffer from quadratic computational complexity and potential instability in high-noise scenarios.

The Selector-Enhancer addresses this by treating attention as a dynamically allocated resource. By routing different regions of a noisy spectrogram to the most appropriate attention mechanism, the model optimizes both performance and efficiency.

The Challenge of Attention in SE

Standard SE models typically employ a uniform processing strategy. However, natural speech varies in complexity. Applying heavy non-local operations to simple, low-noise segments is computationally wasteful. Furthermore, in highly corrupted regions, global attention might inadvertently correlate noise patterns with speech, degrading quality. There is a clear need for a "smart router" that can distinguish between regions requiring local focus and those needing broader context.

The Selector-Enhancer Framework

The proposed architecture consists of two primary components: a dual-attention backbone and an intelligent feature-filter.

Dual-Attention Backbone

The backbone is composed of $N$ dynamic blocks. Each block provides two optional processing paths:

  1. Local Attention Path: Focuses on channel-wise selection and local receptive fields to preserve fine-grained harmonic structures.
  2. Non-local Attention Path: Computes mutual similarity across the entire feature map to capture global context and long-term dependencies.

Feature-Filter and Dynamic Routing

The "brain" of the system is the Feature-Filter, a sub-network utilizing CNNs and LSTMs. It predicts a low-resolution "dispatch policy mask" for each input. This mask dictates which spectrogram regions are processed by the local path and which are sent to the non-local path, effectively acting as a dynamic router.

Reinforcement Learning for Selection

Since the selection process involves discrete, non-differentiable choices, the authors employ the REINFORCE algorithm. A difficulty-regulated reward function is used to train the Feature-Filter, balancing three critical factors:

  • Enhancement Performance: Measured via L2 loss between enhanced and clean speech.
  • Computational Complexity: A penalty term discourages the over-use of the resource-intensive non-local path.
  • Task Difficulty: A regulated term adjusts rewards based on input SNR, ensuring the model learns effectively across varying noise levels.

Experimental Evaluation

The model was rigorously tested on multiple benchmarks, including VoiceBank + DEMAND for standard metrics and the large-scale AVSpeech + AudioSet (4,700 hours) to test scalability. Additionally, WSJ0 SI-84 + NOISEX92 was used to evaluate generalization to unseen speakers and non-stationary noises.

Results and Discussion

The Selector-Enhancer demonstrates superior performance-to-efficiency ratios. On the VoiceBank+DEMAND benchmark, it achieved a PESQ of 3.09 and an STOI of 93.41%. Notably, it required only 2.19 GFLOPs, a significant reduction compared to models like SA-TCN (8.85 GFLOPs), while maintaining a real-time factor (RTF) of 0.63.

Spectrogram visualization comparing noisy speech (a) with various enhancement results (b-f). The dashed boxes highlight regions where the dynamic selection of attention mechanisms effectively preserves speech components while suppressing noise.

Visualization of the policy masks and enhanced spectrograms confirms that the Feature-Filter learns to isolate corrupted speech components. By sending "easy" regions to local attention and "hard" or context-dependent regions to non-local attention, the model achieves high-fidelity enhancement without redundant computation.

Conclusion

The Selector-Enhancer represents a shift toward more intelligent, resource-aware audio processing. By dynamically selecting between local and non-local attention, the model achieves state-of-the-art results with a fraction of the computational cost of traditional all-access attention models. Future work will focus on optimizing the RL training process and validating performance on specialized edge hardware.

Summary updated

BibTeX
@inproceedings{xu2023selector,
  author = {Xinmeng Xu and Weiping Tu and Yuhong Yang},
  title = {{Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26622},
}
Figure