Abstract
Speech enhancement (SE) requires balancing the capture of global acoustic context with the preservation of local harmonic structures. While non-local attention mechanisms provide global reach, they are often computationally expensive and prone to noise interference. This paper presents the Selector-Enhancer, a novel framework that dynamically selects between local and non-local attention operations at the time-frequency (T-F) unit level. Using a reinforcement learning-based "feature-filter," the model intelligently routes signals to the most effective processing path. Experimental results on VoiceBank+DEMAND and AVSpeech+AudioSet benchmarks demonstrate that the Selector-Enhancer achieves state-of-the-art denoising quality (PESQ 3.37) with significantly lower computational overhead than existing global attention models.
1. Introduction
Modern speech enhancement (SE) systems often struggle with the trade-off between local and non-local processing. Local attention is efficient and robust for capturing harmonic structures but lacks a global view of the speaker's identity or long-term temporal patterns. Conversely, non-local attention captures global similarities but suffers from $O(T^2)$ complexity and can be misled by non-stationary noise.
The Selector-Enhancer addresses this by treating attention selection as a dynamic routing problem. Instead of applying a fixed mechanism across all audio segments, the model evaluates the noise and speech characteristics of specific T-F regions to decide the optimal processing strategy.
2. The Selector-Enhancer Framework
The architecture consists of two primary components: the Enhancer backbone and the Selector module.
2.1 Dual-Attention CNN (The Enhancer)
The Enhancer is a convolutional neural network (CNN) composed of $N$ dynamic blocks. Each block features a shared feature extraction path and two parallel attention branches:
- Local Attention Path: Focuses on immediate T-F neighbors.
- Non-local Attention Path: Computes global correlations across the entire spectrogram.
2.2 Feature-Filter (The Selector)
The Feature-Filter acts as a controller. It processes a low-resolution representation of the input features to generate a policy mask. This mask dictates which attention path each T-F unit should follow. By sparsifying the use of non-local attention, the model maintains high efficiency without sacrificing performance.
3. Reinforcement Learning for Dynamic Selection
Since the selection process involves discrete choices, the authors utilize the REINFORCE algorithm for training. A key innovation is the difficulty-regulated reward system. This reward function balances three objectives:
- Reconstruction Accuracy: Minimizing L2 loss between enhanced and clean speech.
- Computational Efficiency: Penalizing excessive use of the non-local path.
- Task Difficulty: Adjusting rewards based on the inherent noise level of the segment, ensuring the model is not unfairly penalized in highly corrupted environments.
4. Experimental Evaluation
The model was evaluated on the VoiceBank + DEMAND and AVSpeech + AudioSet datasets. It consistently outperformed recent baselines, including CRN, CNN-NL, and SA-TCN.
As shown in the figure above, the Selector-Enhancer (f) recovers harmonic structures more accurately than baseline methods (c-e), particularly in the regions highlighted by the dashed boxes. The model achieves a PESQ score of 3.37 and a STOI of 94.36% while maintaining a Real-Time Factor (RTF) of 0.63.
5. Conclusion
The Selector-Enhancer demonstrates that dynamic, context-aware routing is a powerful paradigm for speech enhancement. By selectively applying non-local attention only where it provides clear benefits, the model achieves superior denoising quality and efficiency. Future work will explore extending this dynamic selection logic to other architectural components, such as adaptive kernel sizes.
Summary updated