Multi-Channel Speech Enhancement Using Graph Neural Networks

Panagiotis Tzirakis, Anurag Kumar, Jacob Donley

TL;DR

This paper introduces a novel approach to multi-channel speech enhancement by treating microphone channels as nodes in a dynamic graph. By integrating Graph Convolutional Networks (GCNs) into a U-Net architecture, the model learns complex spatial correlations between microphones on the fly, significantly outperforming traditional beamforming-inspired deep learning methods across various array geometries.

What Problem Does This Paper Address?

In noisy and reverberant environments—the classic "cocktail party" scenario—extracting a clean target voice from multiple microphone feeds is a major challenge. Traditional "beamforming" techniques rely on knowing the exact geometry of the microphone array and the position of the speaker, which is often impossible in real-world conditions.

While deep learning has improved this field, most existing models either try to predict traditional beamforming weights or use standard convolutional layers that assume a fixed spatial relationship between microphones. This paper addresses the need for a more flexible, data-driven way to capture spatial correlations between audio channels, especially when the microphone arrangement is irregular or unknown.

Methods

The authors propose a framework that combines the structural strengths of U-Nets with the relational power of Graph Neural Networks (GNNs).

  1. Audio Representation: Each microphone channel's signal is converted into a complex spectrogram (STFT), providing both magnitude and phase information.
  2. Encoder: A shared encoder (part of a U-Net) extracts high-level features from each channel independently.
  3. Dynamic Graph Construction: This is the "secret sauce." Instead of a fixed grid, the model treats each microphone as a node in a graph. It learns an adjacency matrix—essentially calculating how "similar" or "relevant" each microphone is to the others for a specific audio sample.
  4. Graph Convolutional Network (GCN): In the bottleneck (embedding space) of the U-Net, a GCN aggregates information across these nodes. This allows the network to "communicate" spatial features between microphones.
  5. Decoder & Attention: The decoder reconstructs the signals, and an attention layer performs a weighted sum of these outputs.
  6. Complex Ratio Mask (CRM): The final output is a mask applied to a reference microphone to produce the clean speech estimate.

The model is trained using a multi-objective loss function that considers both the magnitude of the spectrogram and the raw time-domain signal.

Datasets

The researchers used a combination of real speech and simulated environments:

  • LibriSpeech: A standard corpus of clean English speech used for the target signals.
  • AudioSet: Used to provide a wide variety of background noise signals.
  • Simulated Room Acoustics: The authors simulated three types of microphone configurations: linear, circular, and distributed (randomly placed). They tested setups with 2 and 4 microphones in rooms with a reverberation time ($RT_{60}$) of 0.5 seconds.

Key Insights and Ideas

  • Non-Euclidean Perspective: Most audio processing treats channels as a simple list or a grid. By viewing them as a graph, the model can handle any number of microphones in any arrangement (distributed arrays) without needing to be redesigned.
  • Learning the Topology: The graph isn't hard-coded. Because the adjacency matrix is learned from the observations, the network can adapt its "spatial filtering" strategy based on the specific noise and reverberation of the current scene.
  • GCNs as Learnable Beamformers: The GCN layers effectively replace traditional, rigid signal processing heuristics with a learnable mechanism that propagates messages between microphones to find the cleanest signal components.

Key Results

The proposed GCN-based model consistently outperformed the state-of-the-art baseline (CRNN-C) across all metrics: STOI (intelligibility), PESQ (quality), and SDR (distortion).

  • Low SNR Performance: In extremely noisy conditions (-7.5 dB SNR), the model achieved a massive 9 dB improvement in Signal-to-Distortion Ratio (SDR) over the noisy baseline.
  • Ablation Success: Removing the GCN component led to a noticeable drop in performance (e.g., SDR dropped from 7.73 to 7.05 on the development set), proving that the graph-based spatial learning is doing the heavy lifting.
  • Robustness: The model maintained high performance regardless of whether the microphones were in a neat line, a circle, or scattered randomly around the room.

Limitations and Future Work

  • Interpretability: While the model works well, the authors note that future work could involve a more detailed "look under the hood" to see exactly which nodes and edges the model prioritizes in different acoustic scenes.
  • End-to-End Waveforms: Currently, the model operates on STFT spectrograms. The authors suggest that moving to an entirely end-to-end approach using raw waveforms (avoiding the STFT step) could be a promising next step.
  • Real-time Constraints: The paper focuses on accuracy; future iterations would need to evaluate the computational latency for real-time applications like hearing aids or smart speakers.

Summary updated

Abstract

Multi-channel speech enhancement aims to extract clean speech from a noisy mixture using signals captured from multiple microphones. Recently proposed methods tackle this problem by incorporating deep neural network models with spatial filtering techniques such as the minimum variance distortionless response (MVDR) beamformer. In this paper, we introduce a different research direction by viewing each audio channel as a node lying in a non-Euclidean space and, specifically, a graph. This formulation allows us to apply graph neural networks (GNN) to find spatial correlations among the different channels (nodes). We utilize graph convolution networks (GCN) by incorporating them in the embedding space of a U-Net architecture. We use LibriSpeech dataset and simulate room acoustics data to extensively experiment with our approach using different array types, and number of microphones. Results indicate the superiority of our approach when compared to prior state-of-the-art method.
BibTeX
@inproceedings{tzirakis2021multi,
  author = {Panagiotis Tzirakis and Anurag Kumar and Jacob Donley},
  title = {{Multi-Channel Speech Enhancement Using Graph Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Txr85TlCoQ2U},
}

Loading related content...

Figure