TL;DR
This paper introduces a "Hierarchical Representation" (HR) for target speaker extraction, which combines local and global speaker features to improve accuracy. By fusing these features across multiple layers of the separation network, HR significantly outperforms existing methods and even won the ICASSP 2023 Deep Noise Suppression Challenge.
What Problem Does This Paper Address?
Target speaker extraction aims to isolate a specific person's voice from a mixed audio signal, using a short "anchor" utterance from the target speaker as a guide. Current deep learning approaches typically rely on speaker embeddings derived from this anchor. However, these embeddings are often simple, dense vectors (e.g., 1x1024) that can be challenging for the separation network to effectively utilize. This limits the precision of target extraction. The paper addresses this by proposing a more sophisticated, hierarchical representation of the speaker's voiceprint that leverages both fine-grained local features and broader global features.
Methods
The core innovation is the Hierarchical Representation (HR), which combines two types of speaker features:
- Local Speaker Features: These are extracted using an Attentive Recurrent Network (ARN) applied to the anchor utterance. The ARN processes the anchor along the frequency axis, and its outputs, along with the outputs from subsequent convolutional layers, form the local features. These are trained jointly with the main separation network.
- Global Speaker Features: These are derived using a pre-trained ECAPA-TDNN model, a state-of-the-art speaker verification network. This provides a more discriminative, high-level representation of the target speaker.
The HR-TSE system integrates these features into a speech separation network (based on a CRN structure with ARN modules). The local features are fed layer-by-layer into the encoder, while the global features are multiplied with the input of the temporal modeling layer in the decoder. This multi-level fusion is designed to enhance the separation network's ability to pinpoint the target speaker. The system also employs a combination of loss functions: SI-SNR loss for time-domain signal quality, and RI+Mag loss for recovering complex spectral information.
Datasets
- Libri-2talker: Used for training and evaluation. This dataset consists of 127,056 utterances from 1172 speakers for training, and 2344 utterances for development. The evaluation set includes 6000 test utterances.
- DNS2023 blind test set: Used for evaluating performance on real-world recordings, sourced from DEMAND, Freesound, and AudioSet.
Key Insights and Ideas
- Beyond Simple Embeddings: The paper convincingly argues that simple, dense speaker embeddings are a bottleneck. A richer, hierarchical representation is crucial for better target speaker extraction.
- Synergy of Local and Global: Combining fine-grained local features (like pitch, useful for distinguishing speakers with different characteristics) with robust global features (like those from ECAPA-TDNN) leads to superior performance.
- Layered Integration: The proposed method's effectiveness stems from integrating these hierarchical features across multiple layers of the separation network, not just at a single point.
- Winning Formula: The HR approach, when applied to existing strong models like TEA-PSE, significantly boosts their performance, as evidenced by their victory in the ICASSP 2023 DNS Challenge.
Key Results
- State-of-the-Art Performance: The proposed HR-TSE system outperforms several strong baselines (Voicefilter, GatedCRN, TEA-PSE, Speakerfilter) on the Libri-2talker dataset across multiple metrics (SI-SNR, STOI, ESTOI, PESQ).
- Significant Improvements: For example, HR-TSE achieved a 0.493dB higher SI-SNR than TEA-PSE, and improved the overall SI-SNR by 14.446 dB compared to the raw mixture.
- Ablation Study: Using both local and global features (HR) significantly outperforms using either feature type alone.
- Challenge Success: The HR methodology was instrumental in the winning submission for the ICASSP 2023 DNS Challenge, demonstrating its practical effectiveness.
Limitations and Future Work
- Computational Cost: While the paper mentions that HR-equipped models have smaller parameter counts, the hierarchical processing and fusion might introduce some computational overhead compared to simpler methods. This is not explicitly detailed as a limitation but is an inherent trade-off.
- Future Directions: The authors suggest that this hierarchical representation is a general concept that can be applied to various speaker extraction models, implying potential for further exploration and integration with other architectures. The success in the DNS challenge also points towards its utility in real-world noisy environments.
Summary updated