Ripple sparse self-attention for monaural speech enhancement

Qiquan Zhang, Hongxu Zhu, Qi Song, Xinyuan Qian, Zhaoheng Ni, Haizhou Li

Abstract

Transformers have revolutionized speech processing, yet their application in speech enhancement is often hindered by the quadratic computational complexity of standard self-attention and a lack of inherent locality bias. This paper presents Ripple Sparse Self-Attention, a novel mechanism that combines fine-grained local attention with coarse-grained dilated global attention. This "ripple" pattern allows the model to capture both short-term speech correlations and long-term context with significantly lower computational overhead than standard Transformers. Experimental results demonstrate that Ripple Attention outperforms state-of-the-art models like SepFormer in both speech quality and intelligibility while maintaining superior efficiency.

Introduction

Speech enhancement—the task of isolating clean speech from background noise—is a fundamental component of modern communication systems, from hearing aids to mobile telephony. While Transformer architectures have significantly improved performance in this domain, they face two primary challenges. First, the computational cost of standard self-attention grows quadratically ($O(L^2)$) with the length of the audio sequence, making the processing of long recordings prohibitively expensive. Second, standard self-attention treats all time frames equally, failing to exploit the strong local correlations inherent in speech signals, where neighboring frames are typically more relevant than distant ones.

Existing solutions, such as "dual-path" transformers or blockwise attention, often rely on complex chunking strategies that can introduce artifacts at boundaries. To address these limitations, we propose Ripple Sparse Self-Attention, a continuous processing approach that mimics the behavior of a ripple in a pond: dense near the center and increasingly sparse as it radiates outward.

The Ripple Attention Mechanism

The core innovation of this work is the sparsification of the attention matrix through a two-tiered strategy:

  1. Local Band Attention: For every frame, the model attends to its immediate neighbors within a small window ($w$). This ensures the capture of fine-grained, high-resolution local dependencies essential for modeling the phonetic structure of speech.
  2. Dilated Global Attention: To maintain a global context without the quadratic cost, the model samples distant frames outside the local window using a dilation rate ($d$). By attending to every $n$-th frame (e.g., every 16th or 24th), the model gains a coarse-grained "sketch" of the entire sequence.

Architecture

The proposed architecture utilizes 1D-convolutional layers for initial feature projection, followed by a stack of $B$ Ripple Attention blocks. Each block consists of Multi-Head Ripple Attention and a Feed-Forward Network (FFN). The model is trained using two common objectives: the Ideal Ratio Mask (IRM) and the Phase-Sensitive Mask (PSM).

Experimental Setup

Datasets

The model was trained and evaluated using a diverse range of acoustic environments:

  • Clean Speech: 28,539 utterances from the Librispeech (train-clean-100) dataset.
  • Noise Sources: A comprehensive collection of 6,909 noise recordings from MUSAN, QUT-NOISE, and Urban Sound.
  • Evaluation: Testing was conducted on 200 noisy samples featuring unseen noises (e.g., F16 jet noise, factory sounds) at Signal-to-Noise Ratios (SNRs) ranging from -5 dB to 15 dB.

Metrics

Performance was assessed using standard objective measures: Perceptual Evaluation of Speech Quality (PESQ), Extended Short-Time Objective Intelligibility (ESTOI), and composite metrics including CSIG (signal quality), CBAK (background intrusiveness), and COVL (overall quality).

Results and Discussion

Ripple Attention consistently outperformed both standard Transformers and the state-of-the-art SepFormer across all evaluated metrics.

Speech Quality and Intelligibility

At a 5 dB SNR using the IRM target, Ripple Attention achieved a PESQ score of 2.65, a significant improvement over the noisy baseline (1.84) and full self-attention (2.51). Similarly, the model reached an ESTOI of 76.24%, compared to 73.87% for full attention. These results suggest that the ripple pattern effectively captures the necessary information for high-quality speech reconstruction.

Computational Efficiency

Theoretical analysis of Multiply-Accumulate operations (MACs) indicates that Ripple Attention scales much more efficiently than full attention as sequence length increases. By avoiding the overlapping "chunks" required by models like SepFormer, Ripple Attention provides a more streamlined and artifact-free processing pipeline.

Layer-Specific Optimization

An interesting finding during development was the effectiveness of applying local band attention specifically to the lower layers of the Transformer. These layers naturally gravitate toward modeling short-term dependencies, and the ripple structure aligns perfectly with this inductive bias.

Conclusion and Future Work

Ripple Sparse Self-Attention offers a compelling "sweet spot" between computational efficiency and enhancement performance. By prioritizing local resolution while maintaining a global overview, it overcomes the primary weaknesses of standard Transformers in speech processing.

Future research will focus on extending this mechanism to multi-channel (microphone array) setups and optimizing the ripple hyperparameters ($w$ and $d$) for varying audio lengths. Additionally, further study is required to evaluate real-time latency on edge devices, a critical requirement for practical deployment in hearing aids and mobile hardware.

Summary updated

Abstract

The use of Transformer represents a recent success in speech enhancement. However, as its core component, self-attention suffers from quadratic complexity, which is computationally prohibited for long speech recordings. Moreover, it allows each time frame to attend to all time frames, neglecting the strong local correlations of speech signals. This study presents a simple yet effective sparse self-attention for speech enhancement, called ripple attention, which simultaneously performs fine- and coarse-grained modeling for local and global dependencies, respectively. Specifically, we employ local band attention to enable each frame to attend to its closest neighbor frames in a window at fine granularity, while employing dilated attention outside the window to model the global dependencies at a coarse granularity. We evaluate the efficacy of our ripple attention for speech enhancement on two commonly used training objectives. Extensive experimental results consistently confirm the superior performance of the ripple attention design over standard full self-attention, blockwise attention, and dual-path attention (Sep-Former) in terms of speech quality and intelligibility.
BibTeX
@inproceedings{zhang2023ripple,
  author = {Qiquan Zhang and Hongxu Zhu and Qi Song and Xinyuan Qian and Zhaoheng Ni and Haizhou Li},
  title = {{Ripple sparse self-attention for monaural speech enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096693},
}
Figure