Introduction
Speech separation, often referred to as the "cocktail party problem," remains a significant challenge in signal processing, particularly in noisy or reverberant environments. Traditional "blind" separation methods frequently struggle with determining the number of speakers and maintaining permutation consistency. While visual cues like lip movements are noise-immune and provide a strong prior, most existing audio-visual systems operate in the Time-Frequency (TF) domain. These methods often suffer from poor phase reconstruction. This paper proposes a novel time-domain architecture that integrates lip-movement embeddings directly into a TasNet-based framework, bypassing the phase-estimation problem and achieving substantial performance gains.
Methods: The Time-Domain Architecture
The proposed system generalizes the Time-domain Audio Separation Network (TasNet) to incorporate a visual stream. The architecture is composed of four primary components:
- Audio Encoder/Decoder: Instead of relying on Short-Time Fourier Transforms (STFT), the model employs a 1D convolutional layer to learn an optimized representation of the raw audio waveform. A corresponding deconvolutional layer (decoder) reconstructs the separated audio.
- Video Encoder: A 3D convolutional layer followed by an 18-layer ResNet extracts "lip embeddings" from video frames. These embeddings are designed to capture phonetic information from the speaker's mouth movements.
- Separation Network: Utilizing stacked temporal dilated convolutional blocks (Conv-TasNet style), this network fuses encoded audio and video embeddings via concatenation and projection. It predicts a mask specifically for the target speaker.
- Phonetic Pre-training: To ensure the video encoder captures relevant speech features, it is pre-trained on lip-reading tasks (predicting words or phonemes) rather than being trained from scratch on the separation task.
Datasets
The model was trained and evaluated using the following datasets:
- LRS2 (Oxford-BBC Lip Reading Sentences 2): The primary dataset, containing thousands of sentences from BBC television. The researchers simulated 2-speaker and 3-speaker mixtures for evaluation.
- LRW (Lip Reading in the Wild): Used for pre-training the lip embedding extractor on a word-level classification task.
Key Insights and Design Choices
Several critical design choices contributed to the model's success:
- Time Domain Superiority: By operating directly on waveforms, the model avoids the "phase problem" inherent in STFT-based methods.
- Phoneme Embeddings: Comparing different visual targets, the authors found that context-independent phoneme (CI-phone) embeddings provided the optimal balance of granularity and complexity compared to word-level or context-dependent phoneme targets.
- Global Layer Normalization (gLN): The researchers found that replacing standard Batch Normalization (BN) with Global Layer Normalization significantly boosted performance, as gLN better handles the varying scales of audio signals.
Key Results
The model's performance was measured using Scale-invariant Source-to-Noise Ratio (Si-SNR):
- Audio-Visual vs. Audio-Only: The audio-visual model outperformed the audio-only Conv-TasNet by approximately 2.5dB on 2-speaker mixtures and 3.7dB on 3-speaker mixtures.
- Time-Domain vs. Frequency-Domain: It surpassed its frequency-domain counterpart (Conv-FavsNet) by 2.7dB and 3.2dB for 2 and 3 speakers, respectively.
- Optimal Performance: The best configuration, utilizing multi-speaker training, achieved an Si-SNR of 14.02dB for 2-speaker mixtures and 9.92dB for 3-speaker mixtures.
Conclusion and Future Work
While the proposed model demonstrates significant improvements, it relies on simulated mixtures and requires a clear view of the speaker's mouth. Future research could focus on testing the system in more complex, real-world acoustic environments (e.g., moving speakers) and improving robustness to visual occlusions or low-light conditions. Additionally, exploring more computationally efficient fusion mechanisms could help mitigate the increased resource demands of the dual-stream architecture.
Summary updated