Abstract
Modern AI architectures are heavily influenced by the "Attention is All You Need" paradigm. However, this study demonstrates that for speech enhancement (SE) tasks, Recurrent Neural Networks (RNNs) may be superior to attention mechanisms. By replacing attention modules with RNNs in two state-of-the-art models—MTFAA and CMGAN—the researchers observed consistent performance gains across all major speech quality and intelligibility metrics, suggesting that the local inductive bias of RNNs is better suited for the physical properties of speech.
Introduction
Speech Enhancement (SE) aims to isolate clean speech from noisy and reverberant environments. Recently, the Attention Mechanism has become the dominant architecture in SE due to its ability to model long-range dependencies. However, speech signals are inherently local; information at a specific time-frequency point is most strongly influenced by its immediate neighbors. This paper investigates whether the global nature of standard attention is actually a disadvantage compared to the sequential, local-bias nature of RNNs.
Methodology
The researchers conducted a comparative study by modifying two top-performing SE models:
- MTFAA (Multi-scale Temporal Frequency Convolutional Network): The original axial attention module was replaced with a Dual Path RNN (DPRNN), resulting in MTFAA-DPRNN.
- CMGAN (Conformer-based Metric-GAN): The frequency-domain attention modules were replaced with conventional RNNs, resulting in CMGAN-FRNN.
To ensure a fair comparison, training pipelines, datasets, loss functions, and optimizers were kept identical. The models were trained using clean speech from DNS4 and noise from DCASE, with simulated reverberation. Testing was conducted across a wide Signal-to-Noise Ratio (SNR) range from -15 dB to 15 dB.
Key Results
The experimental results consistently favored the RNN-based variants. As shown in the performance tables below, the "brain transplant" of replacing attention with RNNs led to improvements across every evaluated metric.
For the MTFAA model, the average PESQ score increased from 1.678 to 1.839, while the Signal-to-Distortion Ratio (SDR) saw a significant jump from 5.829 dB to 7.250 dB. Similarly, the CMGAN-FRNN outperformed the original CMGAN in PESQ (1.886 vs. 1.817) and other metrics.
Composite scores—CSIG (signal quality), CBAK (background noise intrusiveness), and COVL (overall effect)—also showed marked improvements. For instance, the COVL for MTFAA improved from 2.154 to 2.364 with the inclusion of DPRNN.
Discussion and Conclusion
The findings suggest that the inductive bias of RNNs aligns better with the sequential and local nature of speech signals than global attention mechanisms. This serves as a reminder that popular architectures are not always optimal for every signal processing task.
While the results are compelling, several areas remain for future exploration. The study did not provide a detailed analysis of computational complexity (FLOPs) or inference speed, which are critical for real-time deployment. Additionally, the comparison was limited to global/axial attention; "Local Attention" or "Windowed Attention" might provide a middle ground. Future research could also investigate if these findings generalize to other architectures, such as diffusion-based models.
Summary updated