Abstract
Nowadays, there is a strong need to deploy target speaker separation (TSS) models on mobile devices with limited model size and computational complexity. To address this, we introduce LibriPhone, a dual-channel dataset recorded using physical artificial heads and mobile microphones to mimic real-world scenarios. We propose LSTM-Former, a lightweight time-frequency domain separation model based on an LSTM framework. Experimental results on LibriPhone demonstrate that the dual-channel LSTM-Former outperforms its single-channel counterpart by 25%, providing a feasible and efficient solution for mobile TSS tasks.
Introduction: The "Cocktail Party" on Mobile Devices
The "Cocktail Party Problem"—isolating a single voice in a noisy environment—remains a significant hurdle for mobile voice communication. While deep learning models have shown promise, two primary obstacles hinder their deployment:
- Resource Constraints: High-performance models are often too computationally expensive for mobile CPUs and batteries.
- The Simulation Gap: Training on purely simulated data fails to account for real-world acoustics like reverberation and microphone-specific delays.
Proposed Method: LSTM-Former
The authors propose LSTM-Former, a lightweight model designed for the time-frequency domain. By processing magnitude spectrograms rather than raw waveforms, the model reduces input dimensionality and computational load.
Key Architectural Features:
- Dual-Channel Processing: The model leverages audio from two microphones. It extracts features from both channels, concatenates them with a speaker embedding (the target's vocal "fingerprint"), and predicts a mask to isolate the desired speech.
- Efficient Backbone: Instead of heavy self-attention mechanisms, it utilizes a single-direction LSTM combined with skip connections and Layer Normalization to maintain performance while minimizing latency.
- GCC Alignment: To handle physical recording delays that can disrupt training, Generalized Cross-Correlation (GCC) is used to align signals, ensuring stable convergence during the optimization of the SI-SNR loss.
The LibriPhone Dataset
To bridge the simulation gap, the researchers created LibriPhone. Unlike standard datasets that digitally mix audio, LibriPhone was recorded by playing LibriSpeech samples through two professional artificial heads in a physical room. The recordings were captured using the built-in microphones of an Android smartphone, capturing authentic spatial cues and microphone characteristics.
Experimental Results
The models were evaluated based on Signal-to-Distortion Ratio (SDR) improvement and computational efficiency.
Key Findings:
- Superiority of Dual-Channel: The dual-channel LSTM-Former achieved a 7.71 dB SDR improvement, representing a 25% relative gain over the single-channel version.
- Efficiency for Mobile: With only 1.57M parameters and 154.29M MACs, the model is highly optimized for real-time execution on mobile hardware.
- Spatial Cues are Vital: Increasing the size of a single-channel model provided negligible gains compared to the jump in performance seen when adding a second channel, proving that spatial information is the primary driver of separation quality.
Conclusion
LSTM-Former offers a practical path toward high-quality speaker separation on mobile devices. By combining a lightweight architecture with real-world dual-channel data, the model overcomes the limitations of simulated training. Future work will explore more complex scenarios involving multiple interferers and varying device orientations to further enhance robustness in diverse environments.
Summary updated