Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation

Wupeng Wang, Zexu Pan, Xinke Li, Shuai Wang, Haizhou LiIEEE Transactions on Audio Speech and Language Processing 2025 · DOI

TL;DR

This paper introduces a novel "Causal Self-Supervised Pretrained" (CSP) frontend that significantly improves real-time speech separation by teaching causal models to "predict" future audio information during training. This approach bridges the performance gap between offline and real-time separation systems, leading to higher quality separation with minimal latency.

What Problem Does This Paper Address?

Speech separation (SS) aims to disentangle a mixed audio signal into individual speaker streams. While offline methods achieve good results, they are unsuitable for real-time applications like phone calls or hearing aids, which require processing audio as it arrives (causally). Causal models, by definition, can only use past and present information, which leads to a performance drop compared to offline models that can "see" the entire audio segment. This paper tackles the "causal separation problem" – how to make causal models perform as well as offline models by implicitly incorporating future context during training without violating the causal constraint during inference.

Methods

The core of this work is a new Causal Self-Supervised Pretrained (CSP) frontend. This frontend acts as a feature extractor that learns to generate "predictive patterns" from raw audio mixtures. It uses a causal convolutional encoder to process the input and a transformer decoder to generate these patterns. The key innovation lies in two self-supervised pretext tasks:

  • Autoregressive Hybrid Prediction (AHP): This task trains the frontend to predict future audio information. It combines "top-down" prediction (predicting future latent embeddings) and "bottom-up" prediction (using future latent embeddings to predict current patterns), aiming to capture both future context and maintain audio coherence.
  • Contextual Knowledge Distillation (CKD): This task further enhances the predictive patterns by distilling knowledge from a pre-trained, non-causal "teacher" model (DIP frontend). This helps the causal frontend capture high-level contextual abstractions that are crucial for robust separation.

These two tasks are combined into a multi-task loss to train the CSP frontend. Once trained, this frontend is frozen and used to extract features for a downstream causal speech separation model.

Datasets

The paper uses several datasets for pretraining and evaluation:

  • Pretraining:
    • LM2Mix: Simulated multi-speaker mixtures based on LibriSpeech.
  • Vox2Mix: Simulated multi-speaker mixtures based on VoxCeleb2, known for its diversity in accents and acoustic conditions.
  • REAL-M: Real-world recordings of multi-speaker mixtures without target reference speech.
  • Evaluation:
    • LM2Mix test set: For in-domain evaluation.
  • Vox2Mix and LRS2Mix test sets: For out-of-domain evaluation, testing generalization.
  • REAL-M test set: For evaluating performance on real-world, unseen data.

Key Insights and Ideas

  • Predictive patterns are key to causal separation: The paper strongly suggests that teaching causal models to predict future audio information during training is a highly effective way to overcome the performance limitations imposed by the causal constraint.
  • Human auditory system inspiration: The approach is inspired by how humans naturally develop skills to separate sounds in complex environments, suggesting that learning predictive patterns is a fundamental aspect of auditory processing.
  • Bridging the offline-online gap: The CSP frontend successfully narrows the performance gap between powerful offline SS models and their causal counterparts, making real-time SS much more viable.
  • Domain adaptation through pretraining: Pretraining on diverse datasets (synthetic and real) allows the CSP frontend to learn generalizable features that improve performance even on unseen, out-of-domain datasets.
  • Complementary nature of AHP and CKD: The AHP task focuses on temporal prediction, while CKD focuses on contextual understanding. Combining them leads to superior results, indicating that both aspects are vital for robust feature extraction.

Key Results

  • Significant SI-SDRi improvements: When integrated with the causal ConvTasNet model, the CSP frontend achieved a substantial improvement of 2.14 dB SI-SDRi on the in-domain LM2Mix test set compared to a baseline without a pretrained frontend.
  • Out-of-domain generalization: The CSP frontend demonstrated remarkable generalization, improving SI-SDRi by 3.54 dB on Vox2Mix and 4.58 dB on the unseen LRS2Mix dataset.
  • State-of-the-art performance: The proposed method achieved the highest scores across various metrics (PESQ, STOI, SI-SDRi) when compared to other causal pretrained frontends on the LM2Mix test set.
  • Real-world effectiveness: On the REAL-M dataset, using the CSP frontend significantly reduced Word Error Rate (WER) for ASR systems evaluating the separated speech, demonstrating its utility in practical scenarios. For example, it reduced WER by 14.85% for the ReSepformer model.
  • Minimal latency impact: While the frontend adds some computational cost, the overall real-time factor (RTF) remained below 1.0 for streaming applications.

Limitations and Future Work

  • Computational cost: The frontend increases the number of MACs (Multiply-Accumulate operations), making it more suitable for cloud-based systems than resource-constrained edge devices.
  • Latency: While RTF is good, the frontend does contribute to higher latency, especially in causal models.
  • Future Work: The authors suggest exploring the application of their CSP frontend to other downstream tasks beyond speech separation and investigating its potential for even more complex multi-speaker scenarios. They also highlight the importance of further research into mitigating domain mismatch for real-world applications.

Summary updated

Abstract

Speech separation (SS) seeks to disentangle a multi-speaker speech mixture into single-speaker speech streams. Although SS can be generally achieved using offline methods, such a processing paradigm is not suitable for real-time streaming applications. Causal separation models, which rely only on past and present information,offer a promising solution for real-time streaming. However, these models typically suffer from notable performance degradation due to the absence of future context. In this paper, we introduce a novel frontend that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models during training through predictive patterns. The pretrained frontend employs a transformer decoder network with a causal convolutional encoder as the backbone and is pretrained in a self-supervised manner with two innovative pretext tasks: autoregressive hybrid prediction and contextual knowledge distillation. These tasks allow the model to capture predictive patterns directly from mixtures in a self-supervised manner. The pretrained frontend subsequently serves as a feature extractor to generate high-quality predictive patterns. Comprehensive evaluations on synthetic and real-world datasets are carried out to validate the effectiveness of the proposed pretrained frontend.
BibTeX
@article{wang2025causal,
  author = {Wupeng Wang and Zexu Pan and Xinke Li and Shuai Wang and Haizhou Li},
  title = {{Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation}},
  journal = {IEEE Transactions on Audio Speech and Language Processing},
  year = {2025},
  doi = {10.1109/taslpro.2025.3640969},
}
Figure