Abstract
Speech enhancement (SE) requires balancing the capture of local harmonic structures with long-range non-local dependencies. Traditional approaches often apply fixed attention mechanisms, leading to computational inefficiency or noise-speech correlation errors. We present the Selector-Enhancer, a dual-attention framework that dynamically routes spectrogram regions to either local or non-local attention paths using a reinforcement learning-based feature-filter. Our approach achieves state-of-the-art performance on VoiceBank+DEMAND and large-scale AVSpeech datasets while significantly reducing computational overhead.
Introduction
In speech enhancement, the goal is to extract clean speech from noisy environments. Speech signals are inherently multi-scale, containing brief acoustic events (local) and long-term linguistic or environmental traits (non-local). While Convolutional Neural Networks (CNNs) excel at local pattern recognition, they lack the receptive field for global context. Conversely, Transformers and non-local attention mechanisms capture global dependencies but suffer from quadratic computational complexity and potential instability in high-noise scenarios.
The Selector-Enhancer addresses this by treating attention as a dynamically allocated resource. By routing different regions of a noisy spectrogram to the most appropriate attention mechanism, the model optimizes both performance and efficiency.
The Challenge of Attention in SE
Standard SE models typically employ a uniform processing strategy. However, natural speech varies in complexity. Applying heavy non-local operations to simple, low-noise segments is computationally wasteful. Furthermore, in highly corrupted regions, global attention might inadvertently correlate noise patterns with speech, degrading quality. There is a clear need for a "smart router" that can distinguish between regions requiring local focus and those needing broader context.
The Selector-Enhancer Framework
The proposed architecture consists of two primary components: a dual-attention backbone and an intelligent feature-filter.
Dual-Attention Backbone
The backbone is composed of $N$ dynamic blocks. Each block provides two optional processing paths:
- Local Attention Path: Focuses on channel-wise selection and local receptive fields to preserve fine-grained harmonic structures.
- Non-local Attention Path: Computes mutual similarity across the entire feature map to capture global context and long-term dependencies.
Feature-Filter and Dynamic Routing
The "brain" of the system is the Feature-Filter, a sub-network utilizing CNNs and LSTMs. It predicts a low-resolution "dispatch policy mask" for each input. This mask dictates which spectrogram regions are processed by the local path and which are sent to the non-local path, effectively acting as a dynamic router.
Reinforcement Learning for Selection
Since the selection process involves discrete, non-differentiable choices, the authors employ the REINFORCE algorithm. A difficulty-regulated reward function is used to train the Feature-Filter, balancing three critical factors:
- Enhancement Performance: Measured via L2 loss between enhanced and clean speech.
- Computational Complexity: A penalty term discourages the over-use of the resource-intensive non-local path.
- Task Difficulty: A regulated term adjusts rewards based on input SNR, ensuring the model learns effectively across varying noise levels.
Experimental Evaluation
The model was rigorously tested on multiple benchmarks, including VoiceBank + DEMAND for standard metrics and the large-scale AVSpeech + AudioSet (4,700 hours) to test scalability. Additionally, WSJ0 SI-84 + NOISEX92 was used to evaluate generalization to unseen speakers and non-stationary noises.
Results and Discussion
The Selector-Enhancer demonstrates superior performance-to-efficiency ratios. On the VoiceBank+DEMAND benchmark, it achieved a PESQ of 3.09 and an STOI of 93.41%. Notably, it required only 2.19 GFLOPs, a significant reduction compared to models like SA-TCN (8.85 GFLOPs), while maintaining a real-time factor (RTF) of 0.63.
Visualization of the policy masks and enhanced spectrograms confirms that the Feature-Filter learns to isolate corrupted speech components. By sending "easy" regions to local attention and "hard" or context-dependent regions to non-local attention, the model achieves high-fidelity enhancement without redundant computation.
Conclusion
The Selector-Enhancer represents a shift toward more intelligent, resource-aware audio processing. By dynamically selecting between local and non-local attention, the model achieves state-of-the-art results with a fraction of the computational cost of traditional all-access attention models. Future work will focus on optimizing the RL training process and validating performance on specialized edge hardware.
Summary updated