Abstract
Target speaker separation aims to isolate a specific voice from a mixture of sounds, a task critical for applications ranging from hearing aids to voice-controlled assistants. While deep learning has significantly advanced this field, existing models are often too computationally intensive for real-time use on mobile devices. This paper introduces a lightweight target speaker separation network based on Long Short-Term Memory (LSTM) units and a joint training strategy. By optimizing the network structure and training both speaker registration and separation components simultaneously, the proposed system achieves high performance with a fraction of the parameters of traditional models.
Introduction: The Cocktail Party Problem
The "cocktail party problem"—the human ability to focus on a single speaker in a noisy environment—remains a formidable challenge for machine learning. Current state-of-the-art solutions typically rely on "heavyweight" deep learning models. These models, while accurate, suffer from high latency and large memory footprints, making them impractical for edge devices like smartphones. Furthermore, traditional systems often treat speaker identification (registration) and separation as independent modules, which limits the overall system's optimization potential.
Proposed Methodology
Joint Training Framework
A key innovation of this work is the transition from modular to joint training. Instead of using a static, pre-trained speaker identification model, the researchers developed a pipeline where the speaker registration system and the speaker separation network are optimized together.
As shown in Figure 4, the system takes a reference wave to generate a target embedding (voiceprint) and a mixed wave as input. The joint loss function ($L_{SV}$ and $L_{SS}$) allows backpropagation to update both the registration and separation components. This ensures that the voiceprint embeddings are specifically tuned to aid the separation task, breaking the performance ceiling of modular designs.
Lightweight LSTM Architecture
To address the need for efficiency, the authors designed a hierarchical LSTM-based separation network. The architecture utilizes multiple extraction levels and feature reuse to maintain high audio quality with minimal parameters.
As illustrated in Figure 6, the module incorporates Conditional Layer Normalization (CLN). CLN allows the target speaker's voiceprint to directly modulate the normalization process at various stages of the network. This "steering" mechanism ensures the model remains focused on the target voice throughout the deep layers of the network.
Experimental Results and Insights
The researchers evaluated several versions of their model against the industry-standard Voice-Filter baseline (9.86M parameters):
- LTSS Net (4.64M parameters): This model outperformed the baseline in both Segmental Signal-to-Noise Ratio (SSNR: 12.33 vs 12.17) and Perceptual Evaluation of Speech Quality (PESQ: 2.48 vs 2.44).
- S-LTSS Net (0.55M parameters): This "super-lightweight" variant is 18 times smaller than the baseline. While it showed a slight performance dip compared to the larger LTSS Net, it still provided a massive improvement over unprocessed audio, making it a viable candidate for ultra-low-power microcontrollers.
The results demonstrate that joint optimization and strategic feature reuse allow smaller models to match or exceed the performance of much larger, independently trained systems.
Conclusion and Future Directions
This research provides a blueprint for deploying high-quality speaker separation on mobile hardware. By shrinking the model size to as little as 0.55M parameters without sacrificing significant clarity, the proposed LTSS Net addresses the critical trade-off between performance and efficiency. Future work will focus on moving from the frequency domain to the time domain for even lower latency and exploring multi-channel (dual-microphone) inputs to incorporate spatial cues for better separation in complex acoustic environments.
Summary updated