Lightweight Dual-channel Target Speaker Separation for Mobile Voice Communication

Yuanyuan Bao, Yanze Xu, Na Xu, Wenjing Yang, Hongfeng Li, Shicong Li, Yongtao Jia, Xiang Fei, Jincheng He, Ming Li

Abstract

Nowadays, there is a strong need to deploy target speaker separation (TSS) models on mobile devices with limited model size and computational complexity. To address this, we introduce LibriPhone, a dual-channel dataset recorded using physical artificial heads and mobile microphones to mimic real-world scenarios. We propose LSTM-Former, a lightweight time-frequency domain separation model based on an LSTM framework. Experimental results on LibriPhone demonstrate that the dual-channel LSTM-Former outperforms its single-channel counterpart by 25%, providing a feasible and efficient solution for mobile TSS tasks.

Introduction: The "Cocktail Party" on Mobile Devices

The "Cocktail Party Problem"—isolating a single voice in a noisy environment—remains a significant hurdle for mobile voice communication. While deep learning models have shown promise, two primary obstacles hinder their deployment:

  1. Resource Constraints: High-performance models are often too computationally expensive for mobile CPUs and batteries.
  2. The Simulation Gap: Training on purely simulated data fails to account for real-world acoustics like reverberation and microphone-specific delays.

Proposed Method: LSTM-Former

The authors propose LSTM-Former, a lightweight model designed for the time-frequency domain. By processing magnitude spectrograms rather than raw waveforms, the model reduces input dimensionality and computational load.

Overview of the LSTM-Former architecture for dual-channel target speaker separation.

Key Architectural Features:

  • Dual-Channel Processing: The model leverages audio from two microphones. It extracts features from both channels, concatenates them with a speaker embedding (the target's vocal "fingerprint"), and predicts a mask to isolate the desired speech.
  • Efficient Backbone: Instead of heavy self-attention mechanisms, it utilizes a single-direction LSTM combined with skip connections and Layer Normalization to maintain performance while minimizing latency.
  • GCC Alignment: To handle physical recording delays that can disrupt training, Generalized Cross-Correlation (GCC) is used to align signals, ensuring stable convergence during the optimization of the SI-SNR loss.

The LibriPhone Dataset

To bridge the simulation gap, the researchers created LibriPhone. Unlike standard datasets that digitally mix audio, LibriPhone was recorded by playing LibriSpeech samples through two professional artificial heads in a physical room. The recordings were captured using the built-in microphones of an Android smartphone, capturing authentic spatial cues and microphone characteristics.

Experimental Results

The models were evaluated based on Signal-to-Distortion Ratio (SDR) improvement and computational efficiency.

Performance comparison of single-channel and dual-channel models on the LibriPhone dataset.

Key Findings:

  • Superiority of Dual-Channel: The dual-channel LSTM-Former achieved a 7.71 dB SDR improvement, representing a 25% relative gain over the single-channel version.
  • Efficiency for Mobile: With only 1.57M parameters and 154.29M MACs, the model is highly optimized for real-time execution on mobile hardware.
  • Spatial Cues are Vital: Increasing the size of a single-channel model provided negligible gains compared to the jump in performance seen when adding a second channel, proving that spatial information is the primary driver of separation quality.

Conclusion

LSTM-Former offers a practical path toward high-quality speaker separation on mobile devices. By combining a lightweight architecture with real-world dual-channel data, the model overcomes the limitations of simulated training. Future work will explore more complex scenarios involving multiple interferers and varying device orientations to further enhance robustness in diverse environments.

Summary updated

Abstract

Nowadays, there is a strong need to deploy the target speaker separation (TSS) model on mobile devices with a limitation of the model size and computational complexity. To better perform TSS for mobile voice communication, we first make a dual-channel dataset based on a specific scenario, LibriPhone. Specifically, to better mimic the real-case scenario, instead of simulating from the single-channel dataset, LibriPhone is made by simultaneously replaying pairs of utterances from LibriSpeech by two professional artificial heads and recording by two built-in microphones of the mobile. Then, we propose a lightweight time-frequency domain separation model, LSTM-Former, which is based on the LSTM framework with source-to-noise ratio (SI-SNR) loss. For the experiments on Libri-Phone, we explore the dual-channel LSTMFormer model and a single-channel version by a random single channel of Libri-Phone. Experimental result shows that the dual-channel LSTM-Former outperforms the single-channel LSTMFormer with relative 25% improvement. This work provides a feasible solution for the TSS task on mobile devices, playing back and recording multiple data sources in real application scenarios for getting dual-channel real data can assist the lightweight model to achieve higher performance.
BibTeX
@misc{bao2021lightweight,
  author = {Yuanyuan Bao and Yanze Xu and Na Xu and Wenjing Yang and Hongfeng Li and Shicong Li and Yongtao Jia and Xiang Fei and Jincheng He and Ming Li},
  title = {{Lightweight Dual-channel Target Speaker Separation for Mobile Voice Communication}},
  year = {2021},
  eprint = {2106.02934},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.02934},
}
Figure