Efficient Multi-Channel Speech Enhancement with Spherical Harmonics Injection for Directional Encoding

Jiahui Pan, Pengjie Shen, Hui Zhang, Xueliang Zhang

TL;DR

Researchers have developed a more efficient way to clean up noisy speech recorded by multiple microphones by "injecting" spatial information via the Spherical Harmonics Transform (SHT). Their dual-encoder model outperforms current state-of-the-art benchmarks on speech quality and intelligibility while requiring fewer parameters and less computational power.

What Problem Does This Paper Address?

Multi-channel speech enhancement (MCSE) is the art of separating a target voice from background noise and echo using an array of microphones (think smart speakers or conference systems). The "superpower" of multi-channel systems is their ability to use spatial cues—the tiny differences in timing and volume between microphones that tell us where a sound is coming from.

While deep learning has revolutionized this field, most models simply "stack" the raw audio data (STFT) from every microphone and hope the neural network figures out the geometry on its own. This is computationally expensive and often fails to capture the underlying spatial structure of the soundfield effectively. The authors identify a need for a more concise, mathematically grounded way to represent "space" for the model.

Methods

The core innovation is the use of Spherical Harmonics Transform (SHT) coefficients as auxiliary inputs.

  1. Spherical Harmonics Injection: Instead of just using raw audio, the researchers transform the microphone signals into the spherical harmonic domain. This represents the soundfield as a combination of basis functions on a sphere. It’s a compact way to describe the spatial distribution of sound—lower-order harmonics capture broad patterns, while higher-order ones capture fine details.
  2. Dual-Encoder Architecture: The model features two separate "specialist" encoders:
  3. STFT Encoder: Focuses on the spectral-temporal features (the "what" and "when" of the sound).
  4. SHT Encoder: Focuses on the spatial features (the "where" of the sound).
  5. Parallel Processing & Fusion: The outputs of these two encoders are concatenated and fed into a Bi-LSTM (Bidirectional Long Short-Term Memory) layer to refine the spatial-temporal relationship before a decoder reconstructs the clean speech.
  6. Efficiency: By using "inplace" gated linear units (GLUs), the model maintains a high level of performance without the massive memory overhead typical of complex convolutional networks.

Datasets

The authors evaluated their model using a rigorous simulation setup:

  • Speech: Clean speech from the TIMIT database.
  • Noise: Interference from the DNS Challenge corpus, NOISEX-92, and cafe noises from CHiME3.
  • Environment: They simulated a 9-microphone uniform circular array in rooms with varying levels of reverberation (RT60 ranging from 0.2s to 1.0s) and Signal-to-Noise Ratios (SNR from -6dB to 6dB).

Key Insights and Ideas

  • Spatial Conciseness: SHT coefficients act as a "compressed" version of the spatial environment. This allows the model to understand the geometry of the room without needing to process every raw audio channel simultaneously in a single, massive encoder.
  • The Power of Parallelism: A key "aha" moment came from their ablation study. They found that processing SHT and STFT features in parallel encoders worked significantly better than "serial" processing (concatenating them at the very beginning). Parallel encoding allows the network to learn specialized representations for "sound" and "space" before trying to combine them.
  • Doing More with Less: Remarkably, the proposed model is more efficient than the baseline (IGCRN). It uses 1.82M parameters (vs. 1.89M) and 19.52 GFLOPs (vs. 21.01 GFLOPs) while delivering better results.

Key Results

The model was tested against standard baselines like GCRN and IGCRN using two primary metrics: PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility).

  • Quality Boost: At a very noisy -5dB SNR, the proposed parallel model achieved a PESQ score of 1.89, significantly higher than the baseline IGCRN's 1.74.
  • Intelligibility Gain: In the same -5dB environment, the STOI score (representing how much of the speech is understandable) jumped from 65.97% (baseline) to 71.41%.
  • Robustness: The gains were consistent across all tested reverberation times (0.2s to 0.6s), proving that the SHT injection helps the model "see through" the echo.

Limitations and Future Work

  • Array Geometry: The experiments were conducted on a uniform circular array. While SHT is a general mathematical tool, the paper doesn't deeply explore how the model handles irregular or "ad-hoc" microphone placements (like several phones scattered on a table).
  • Real-World Validation: The results are based on simulated room impulse responses. Testing on real-world recordings with hardware-specific noise and non-ideal microphone responses would be the next logical step.
  • Broader Applications: The authors suggest that this "spatial injection" method could be extended beyond speech enhancement to tasks like spatial audio rendering or source localization.

Summary updated

Abstract

Multi-channel speech enhancement extracts speech using multiple microphones that capture spatial cues. Effectively utilizing directional information is therefore key. Deep learning shows great potential on multi-channel speech enhancement and often takes short-time Fourier Transform (STFT) as inputs directly. To fully leverage the spatial information, we introduce a method using spherical harmonics transform (SHT) coefficients as auxiliary model inputs. These coefficients concisely represent spatial distributions. Specifically, our model has two encoders, one for the STFT and another for the SHT. By fusing both encoders in the decoder to estimate the enhanced STFT, we effectively incorporate spatial context. Evaluations on TIMIT under varying noise and reverberation show our model outperforms established benchmarks. Remarkably, this is achieved with fewer computations and parameters. By leveraging spherical harmonics to incorporate directional cues, our model efficiently improves the performance of the multi-channel speech enhancement.
BibTeX
@inproceedings{pan2024efficient,
  author = {Jiahui Pan and Pengjie Shen and Hui Zhang and Xueliang Zhang},
  title = {{Efficient Multi-Channel Speech Enhancement with Spherical Harmonics Injection for Directional Encoding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=EoATvVQ6NR},
}

Loading related content...

Figure