Dualsep: A Light-Weight Dual-Encoder Convolutional Recurrent Network For Real-Time In-Car Speech Separation

Ziqian Wang, Jiayao Sun, Zihan Zhang, Xingchen Li, Jie Liu, Lei XieSLT 2024 · DOI

TL;DR

This paper introduces DualSep, a lightweight deep learning model for separating individual passenger voices in noisy, multi-microphone in-car environments. By combining traditional signal processing techniques with a novel dual-encoder architecture, DualSep achieves high-quality speech separation with significantly reduced computational cost, making real-time in-car voice interaction more feasible.

What Problem Does This Paper Address?

The increasing sophistication of in-car voice assistants and infotainment systems hinges on accurate understanding of passenger speech. However, the in-car environment presents a formidable challenge for speech separation: multiple passengers speaking simultaneously, often from different locations, combined with the inherent noise and reverberation of a vehicle, and the strict latency and computational constraints of automotive systems. Existing deep learning approaches, while powerful, often struggle with these constraints due to their complexity and reliance on non-causal processing, making them unsuitable for real-time applications. This paper aims to bridge this gap by developing an efficient, real-time speech separation system specifically tailored for the in-car context.

Methods

The proposed DualSep system employs a hybrid approach, integrating Digital Signal Processing (DSP) with a Neural Network (NN) architecture.

  • DSP Preprocessing:
    • Fixed Beamforming (BF): This technique is used to reduce the number of audio channels by focusing on signals from specific directions, thereby lowering computational overhead.
  • Independent Vector Analysis (IVA): Applied after BF, IVA further processes the signals to derive spatial priors, helping to distinguish between sources based on their spatial characteristics.
  • Neural Network Architecture:
    • Dual-Encoder: This is the core innovation. It comprises two parallel encoders:
  • Spatial Encoder: Processes the IVA-enhanced signals to capture spatial cues.
  • Spectral Encoder: Processes the beamformed signals to preserve spectral information. This dual-branch approach allows for a more comprehensive modeling of both spatial and spectral aspects of the speech signals.

  • Triple-Path Modeling Module: This module integrates the outputs of the dual encoders to capture complex dependencies across time, frequency, and spatial dimensions. It uses specialized RNN blocks (F-RNN, S-RNN, T-RNN) to model spectral, spatial, and temporal characteristics, respectively.

  • Decoder: Reconstructs the separated speech signals from the learned latent representations.
  • Streaming and Non-Streaming Modes: The architecture is designed to be flexible, allowing for seamless switching between streaming (real-time) and non-streaming (offline) modes. For streaming, bidirectional RNNs and non-causal convolutions are replaced with their unidirectional and causal counterparts.

Datasets

The authors created simulated multi-speaker in-car datasets by combining:

  • AISHELL-1: A clean speech corpus.
  • DNS Challenge 5: A noise corpus.

They generated multichannel room impulse responses (RIRs) using an image-source method to simulate various cabin sizes and acoustic conditions. The datasets were configured with 1 to 6 passengers speaking simultaneously, with varying signal-to-distortion ratios (SDR). The datasets were split into Train (100K utterances), Dev (5K utterances), and Test (1K utterances) sets, ensuring no overlap in noise types or RIRs between sets for unbiased evaluation.

Key Insights and Ideas

  • The Power of Hybrid DSP+NN: Combining traditional DSP techniques like beamforming and IVA with deep learning proves highly effective. DSP handles initial noise reduction and spatial cue extraction, significantly reducing the burden on the neural network.
  • Dual-Encoder for Complementary Information: Separately encoding spatial and spectral information in parallel branches allows the model to capture richer, complementary features than a single encoder could. This is crucial for disentangling overlapping speech in complex acoustic environments.
  • Lightweight Design for Real-World Deployment: The emphasis on a lightweight architecture (0.83M parameters) and achieving a low Real-Time Factor (RTF) of 0.39 on a standard CPU is critical for practical in-car applications where computational resources are limited.
  • Streaming Performance Close to Offline: The ability to achieve near-offline performance in streaming mode is a significant achievement, enabling real-time voice interaction without substantial quality degradation.

Key Results

  • Superior Performance: DualSep outperforms several state-of-the-art baselines across key metrics like SiSNR (Scale-Invariant Source-to-Noise Ratio), PESQ (Perceptual Evaluation of Speech Quality), and CER (Character Error Rate). For instance, it achieves a SiSNR of 13.50 dB and a CER of 7.65% on the test set.
  • Efficiency: The proposed system is remarkably efficient, boasting only 0.83 million parameters and a Real-Time Factor (RTF) of 0.39 on an Intel Core i7 CPU. This is significantly lower than many competing methods.
  • Ablation Study Highlights:
    • Single encoders (spectral or spatial only) showed a notable performance drop compared to the dual-encoder, underscoring the importance of both spatial and spectral information.
  • Combining BF and IVA as a preprocessing step yielded a significant improvement (nearly 10% reduction in CER) over BF alone, validating the utility of IVA for spatial prior extraction.

Limitations and Future Work

  • Limitations:
    • The paper focuses on separating speech into distinct "speech zones," implying that each zone is primarily occupied by one speaker. Handling scenarios with multiple overlapping speakers within a single zone might require further advancements.
  • While the system is designed for real-time, the exact performance on embedded automotive hardware is not detailed, which is the ultimate deployment target.
  • Future Work:
    • The authors suggest that their system holds promise for real-time communication and offline applications, implying potential extensions to other domains.
  • Further exploration of fusion strategies within the dual-encoder framework could potentially yield even better performance.
  • Investigating the impact of different microphone array configurations and more diverse acoustic environments would be valuable.

Summary updated

BibTeX
@inproceedings{wang2024dualsep,
  author = {Ziqian Wang and Jiayao Sun and Zihan Zhang and Xingchen Li and Jie Liu and Lei Xie},
  title = {{Dualsep: A Light-Weight Dual-Encoder Convolutional Recurrent Network For Real-Time In-Car Speech Separation}},
  booktitle = {SLT},
  year = {2024},
  doi = {10.1109/slt61566.2024.10832223},
}
Figure