Lightweight target speaker separation network based on joint training

Jing Wang, Hanyue Liu, Liang Xu, Wenjing Yang, Weiming Yi, Fang LiuEURASIP 2023 · DOI

Abstract

Target speaker separation aims to isolate a specific voice from a mixture of sounds, a task critical for applications ranging from hearing aids to voice-controlled assistants. While deep learning has significantly advanced this field, existing models are often too computationally intensive for real-time use on mobile devices. This paper introduces a lightweight target speaker separation network based on Long Short-Term Memory (LSTM) units and a joint training strategy. By optimizing the network structure and training both speaker registration and separation components simultaneously, the proposed system achieves high performance with a fraction of the parameters of traditional models.

Introduction: The Cocktail Party Problem

The "cocktail party problem"—the human ability to focus on a single speaker in a noisy environment—remains a formidable challenge for machine learning. Current state-of-the-art solutions typically rely on "heavyweight" deep learning models. These models, while accurate, suffer from high latency and large memory footprints, making them impractical for edge devices like smartphones. Furthermore, traditional systems often treat speaker identification (registration) and separation as independent modules, which limits the overall system's optimization potential.

Proposed Methodology

Joint Training Framework

A key innovation of this work is the transition from modular to joint training. Instead of using a static, pre-trained speaker identification model, the researchers developed a pipeline where the speaker registration system and the speaker separation network are optimized together.

Figure 4: The overall architecture of the proposed joint training system for target speaker separation.

As shown in Figure 4, the system takes a reference wave to generate a target embedding (voiceprint) and a mixed wave as input. The joint loss function ($L_{SV}$ and $L_{SS}$) allows backpropagation to update both the registration and separation components. This ensures that the voiceprint embeddings are specifically tuned to aid the separation task, breaking the performance ceiling of modular designs.

Lightweight LSTM Architecture

To address the need for efficiency, the authors designed a hierarchical LSTM-based separation network. The architecture utilizes multiple extraction levels and feature reuse to maintain high audio quality with minimal parameters.

Figure 6: Detailed structure of the lightweight speech separation module, featuring hierarchical LSTM layers and Conditional Layer Normalization (CLN).

As illustrated in Figure 6, the module incorporates Conditional Layer Normalization (CLN). CLN allows the target speaker's voiceprint to directly modulate the normalization process at various stages of the network. This "steering" mechanism ensures the model remains focused on the target voice throughout the deep layers of the network.

Experimental Results and Insights

The researchers evaluated several versions of their model against the industry-standard Voice-Filter baseline (9.86M parameters):

  • LTSS Net (4.64M parameters): This model outperformed the baseline in both Segmental Signal-to-Noise Ratio (SSNR: 12.33 vs 12.17) and Perceptual Evaluation of Speech Quality (PESQ: 2.48 vs 2.44).
  • S-LTSS Net (0.55M parameters): This "super-lightweight" variant is 18 times smaller than the baseline. While it showed a slight performance dip compared to the larger LTSS Net, it still provided a massive improvement over unprocessed audio, making it a viable candidate for ultra-low-power microcontrollers.

The results demonstrate that joint optimization and strategic feature reuse allow smaller models to match or exceed the performance of much larger, independently trained systems.

Conclusion and Future Directions

This research provides a blueprint for deploying high-quality speaker separation on mobile hardware. By shrinking the model size to as little as 0.55M parameters without sacrificing significant clarity, the proposed LTSS Net addresses the critical trade-off between performance and efficiency. Future work will focus on moving from the frequency domain to the time domain for even lower latency and exploring multi-channel (dual-microphone) inputs to incorporate spatial cues for better separation in complex acoustic environments.

Summary updated

Abstract

Abstract Target speaker separation aims to separate the speech components of the target speaker from mixed speech and remove extraneous components such as noise. In recent years, deep learning-based speech separation methods have made significant breakthroughs and have gradually become mainstream. However, these existing methods generally face problems with system latency and performance upper limits due to the large model size. To solve these problems, this paper proposes improvements in the network structure and training methods to enhance the model’s performance. A lightweight target speaker separation network based on long-short-term memory (LSTM) is proposed, which can reduce the model size and computational delay while maintaining the separation performance. Based on this, a target speaker separation method based on joint training is proposed to achieve the overall training and optimization of the target speaker separation system. Joint loss functions based on speaker registration and speaker separation are proposed for joint training of the network to further improve the system’s performance. The experimental results show that the lightweight target speaker separation network proposed in this paper has better performance while being lightweight, and joint training of the target speaker separation network with our proposed loss function can further improve the separation performance of the original model.
BibTeX
@inproceedings{wang2023lightweight,
  author = {Jing Wang and Hanyue Liu and Liang Xu and Wenjing Yang and Weiming Yi and Fang Liu},
  title = {{Lightweight target speaker separation network based on joint training}},
  booktitle = {EURASIP},
  year = {2023},
  doi = {10.1186/s13636-023-00317-3},
}
Figure