A Novel Temporal Attentive-Pooling based Convolutional Recurrent Architecture for Acoustic Signal Enhancement

Tassadaq Hussain, Wei-Chien Wang, Mandar Gogate, Kia Dashtipour, Yu Tsao, Xugang Lu, Ahsan Adeel, Amir Hussain

Abstract

In acoustic signal processing, target signals carry semantic information encoded in a hierarchical structure of short and long-term contexts. However, background noise distorts these structures non-uniformly, a challenge often ignored by existing deep acoustic signal enhancement (ASE) architectures. To address this, we propose integrating a novel temporal attentive-pooling (TAP) mechanism into a conventional convolutional recurrent neural network, termed TAP-CRNN. This approach considers both global and local attention for ASE tasks. Specifically, a convolutional layer extracts local information, followed by a recurrent neural network (RNN) to characterize temporal contextual information. A novel attention mechanism then contextually processes salient regions of noisy signals. Evaluated on a benchmark infant cry dataset, the TAP-CRNN effectively reduces noise components in unseen background noises at challenging signal-to-noise levels, outperforming several well-known methods.

The Challenge of Acoustic Signal Enhancement

Acoustic Signal Enhancement (ASE) aims to isolate target signals—such as speech or an infant's cry—from background interference like wind, mechanical hums, or chatter. The primary difficulty lies in the fact that noise does not merely overlay the audio; it distorts the hierarchical structure of the sound. Audio signals possess both "local" features (short-term spectral patterns) and "global" features (long-term temporal context).

Most existing deep learning models apply a uniform processing approach, failing to distinguish between salient parts of the audio and irrelevant noise. This often results in "blended" outputs where the target signal remains muddy, particularly in non-stationary environments where noise characteristics change rapidly over time.

The TAP-CRNN Architecture

The proposed Temporal Attentive-Pooling Convolutional Recurrent Neural Network (TAP-CRNN) addresses these limitations through a multi-stage pipeline designed to dissociate signal from noise more effectively.

  1. Feature Extraction: Raw audio is transformed into Log-Power Spectrum (LPS) features via Short-Time Fourier Transform (STFT).
  2. CRNN Backbone: CNN layers extract local spatial-spectral information, while Bidirectional Long Short-Term Memory (LSTM) layers characterize the long-term temporal context.
  3. TAP Mechanism: The core innovation is the Temporal Attentive-Pooling block. It employs Global Attention to identify important frames across the entire sequence and Local Attention to refine feature extraction at specific time steps, capturing short-term saliency.
  4. Reconstruction: The attentive context is merged with the CRNN output to predict the clean signal and noise components separately.
Figure 1: Overview of the proposed TAP-CRNN architecture for acoustic signal enhancement, showing the transition from noisy input to enhanced signal through STFT, the CRNN-TAP backbone, and ISTFT.

Experimental Evaluation

The model was rigorously tested using a benchmark infant cry dataset. The training phase involved 7,200 utterances corrupted by six noise types (e.g., Babble, Pink noise, and cocktail party) at Signal-to-Noise Ratios (SNRs) of -5dB, 0dB, and 5dB. To test generalization, the researchers evaluated the model against "unseen" noise types—stationary Fan noise and non-stationary Song/Female speech—at challenging levels ranging from -6dB to 6dB.

Performance Analysis

The TAP-CRNN was compared against four baselines: standard DNN, CNN, RNN, and a basic CRNN. The results demonstrated a clear advantage for the proposed architecture, particularly in high-noise environments.

Figure 5: Comparative performance of TAP-CRNN against baseline models across four key metrics: (a) SDR, (b) SAR, (c) SIR, and (d) SSNR across different SNR levels.

Key findings include:

  • Superiority at Low SNR: At -6dB, where noise significantly overpowers the target signal, TAP-CRNN achieved a Segmental SNR (SSNR) of 13.375 in stationary noise, outperforming all baselines.
  • Metric Consistency: The model showed consistent gains across Signal Distortion Ratio (SDR), Signal to Interference Ratio (SIR), and Signal to Artifacts Ratio (SAR).
  • Saliency Identification: Spectrogram analysis confirmed that the TAP mechanism correctly identified and restored low-frequency acoustic regions that were nearly buried in noise, which standard CRNNs often misidentified.

Conclusion and Future Directions

By combining local and global attention, the TAP-CRNN preserves the delicate spectral envelope of target signals while ignoring complex background noise. While the results are promising, future work will focus on expanding the dataset beyond the initial five infants to ensure broader generalization. Additionally, the team aims to optimize the model for real-time applications, such as hearing aids, which require extremely low latency (<10ms), and explore multi-modal integration using visual data to further enhance performance in extreme environments.

Summary updated

Abstract

In acoustic signal processing, the target signals usually carry semantic information, which is encoded in a hierarchal structure of short and long-term contexts. However, the background noise distorts these structures in a nonuniform way. The existing deep acoustic signal enhancement (ASE) architectures ignore this kind of local and global effect. To address this problem, we propose to integrate a novel temporal attentive-pooling (TAP) mechanism into a conventional convolutional recurrent neural network, termed as TAP-CRNN. The proposed approach considers both global and local attention for ASE tasks. Specifically, we first utilize a convolutional layer to extract local information of the acoustic signals and then a recurrent neural network (RNN) architecture is used to characterize temporal contextual information. Second, we exploit a novelattention mechanism to contextually process salient regions of the noisy signals. The proposed ASE system is evaluated using a benchmark infant cry dataset and compared with several well-known methods. It is shown that the TAPCRNN can more effectively reduce noise components from infant cry signals in unseen background noises at challenging signal-to-noise levels.
BibTeX
@article{hussain2022novel,
  author = {Tassadaq Hussain and Wei-Chien Wang and Mandar Gogate and Kia Dashtipour and Yu Tsao and Xugang Lu and Ahsan Adeel and Amir Hussain},
  title = {{A Novel Temporal Attentive-Pooling based Convolutional Recurrent Architecture for Acoustic Signal Enhancement}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=EuWzKcyjrV},
}
Figure