Multi-Channel Speech Enhancement using Graph Neural Networks

Panagiotis Tzirakis, Anurag Kumar, Jacob Donley

Abstract

Multi-channel speech enhancement aims to extract clean speech from a noisy mixture using signals captured from multiple microphones. Recently proposed methods tackle this problem by incorporating deep neural network models with spatial filtering techniques such as the minimum variance distortionless response (MVDR) beamformer. In this paper, we introduce a different research direction by viewing each audio channel as a node lying in a non-Euclidean space and, specifically, a graph. This formulation allows us to apply graph neural networks (GNN) to find spatial correlations among the different channels (nodes). We utilize graph convolution networks (GCN) by incorporating them in the embedding space of a U-Net architecture. We use LibriSpeech dataset and simulate room acoustics data to extensively experiment with our approach using different array types, and number of microphones. Results indicate the superiority of our approach when compared to prior state-of-the-art method.

Introduction

In noisy and reverberant environments—the classic "cocktail party" scenario—extracting a clean target voice from multiple microphone feeds is a major challenge. Traditional "beamforming" techniques rely on knowing the exact geometry of the microphone array and the position of the speaker, which is often impossible in real-world conditions. While deep learning has improved this field, most existing models either try to predict traditional beamforming weights or use standard convolutional layers that assume a fixed spatial relationship between microphones. This research addresses the need for a more flexible, data-driven way to capture spatial correlations between audio channels, especially when the microphone arrangement is irregular or unknown.

Proposed Architecture

The framework combines the structural strengths of U-Nets with the relational power of Graph Neural Networks (GNNs). The model treats each microphone as a node in a dynamic graph rather than a fixed grid.

Proposed architecture integrating GCN within a U-Net for multi-channel speech enhancement.
  1. Audio Representation: Each microphone channel's signal is converted into a complex spectrogram (STFT), providing both magnitude and phase information.
  2. Encoder: A shared encoder extracts high-level features from each channel independently.
  3. Dynamic Graph Construction: The model learns an adjacency matrix, calculating the similarity or relevance of each microphone to the others for a specific audio sample.
  4. Graph Convolutional Network (GCN): In the bottleneck (embedding space) of the U-Net, a GCN aggregates information across these nodes, allowing the network to communicate spatial features between microphones.
  5. Decoder & Attention: The decoder reconstructs the signals, and an attention layer performs a weighted sum of these outputs to produce a final mask applied to a reference microphone.

Experimental Setup

The researchers utilized a combination of real speech and simulated environments:

  • LibriSpeech & AudioSet: Used for target signals and a wide variety of background noise signals, respectively.
  • Simulated Room Acoustics: Three types of microphone configurations were simulated: linear, circular, and distributed (randomly placed). Setups with 2 and 4 microphones were tested in rooms with a reverberation time ($RT_{60}$) of 0.5 seconds.

Results and Discussion

The proposed GCN-based model consistently outperformed the state-of-the-art baseline (CRNN-C) across all metrics: STOI (intelligibility), PESQ (quality), and SDR (distortion).

Performance comparison (STOI, PESQ, SDR) across different input SNR levels.

Key findings include:

  • Low SNR Performance: In extremely noisy conditions (-7.5 dB SNR), the model achieved a massive ~9.9 dB improvement in Signal-to-Distortion Ratio (SDR) over the noisy baseline.
  • Robustness: The model maintained high performance regardless of array geometry (linear, circular, or distributed), demonstrating the flexibility of the non-Euclidean graph perspective.
  • Ablation Study: Removing the GCN component led to a drop in SDR from 7.73 to 7.05, proving that graph-based spatial learning is critical to the model's success.

Conclusion

By viewing microphone channels as nodes in a graph, this approach allows for learnable spatial filtering that adapts to the specific noise and reverberation of a scene. Future work will explore end-to-end raw waveform processing and evaluate computational latency for real-time applications like hearing aids or smart speakers.

Summary updated

Abstract

Multi-channel speech enhancement aims to extract clean speech from a noisy mixture using signals captured from multiple microphones. Recently proposed methods tackle this problem by incorporating deep neural network models with spatial filtering techniques such as the minimum variance distortionless response (MVDR) beamformer. In this paper, we introduce a different research direction by viewing each audio channel as a node lying in a non-Euclidean space and, specifically, a graph. This formulation allows us to apply graph neural networks (GNN) to find spatial correlations among the different channels (nodes). We utilize graph convolution networks (GCN) by incorporating them in the embedding space of a U-Net architecture. We use LibriSpeech dataset and simulate room acoustics data to extensively experiment with our approach using different array types, and number of microphones. Results indicate the superiority of our approach when compared to prior state-of-the-art method.
BibTeX
@inproceedings{tzirakis2021multi,
  author = {Panagiotis Tzirakis and Anurag Kumar and Jacob Donley},
  title = {{Multi-Channel Speech Enhancement using Graph Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Txr85TlCoQ2U},
}
Figure