SepALM: Audio Language Models Are Error Correctors for Robust Speech Separation

Zhaoxi Mu, Xinyu Yang, Gang Wang

TL;DR

This paper introduces SepALM, a novel approach to speech separation that leverages audio language models (ALMs) to correct errors in preliminary separated speech within the text domain. By performing error correction in the text domain and then re-synthesizing the speech, SepALM achieves more robust separation in challenging acoustic environments and outperforms existing methods.

What Problem Does This Paper Address?

Speech separation, the task of isolating individual voices from a mixed audio signal (like the "cocktail party problem"), is a well-studied area. However, current state-of-the-art methods often struggle in real-world scenarios characterized by significant noise and reverberation. These challenging conditions can introduce artifacts and distortions into the separated speech, degrading its quality. While some approaches try to mitigate these issues through multi-stage processing or generative models, their effectiveness is often limited by the complexity and variability of real-world acoustic interference, which can go beyond typical training data.

Methods

SepALM proposes a four-component framework: a separator, a corrector, a synthesizer, and an aligner. The core innovation lies in the corrector, which uses an Audio Language Model (ALM) to fix errors in the textual transcription of the preliminary separated speech, rather than directly manipulating the audio. This is motivated by the idea that text is a lower-resolution representation, making error correction simpler and less susceptible to noise.

Key techniques employed include:

  • Single ALM for End-to-End Correction: Instead of a cascaded system of Automatic Speech Recognition (ASR) and Large Language Models (LLMs), SepALM uses a single ALM. This reduces error accumulation and simplifies optimization.
  • Chain-of-Thought (CoT) Prompting: To enhance the ALM's reasoning and prevent hallucinations, a CoT approach is used, breaking down the correction into preliminary transcription and then refinement.
  • Knowledge Distillation: Given the scarcity of annotated data for this specific task, a pre-trained ASR model (Whisper Tiny) is used as a "teacher" to guide the ALM's training.
  • Speech Synthesis: The corrected text transcription is then used to re-synthesize the refined speech. This is done using a neural codec language model, treating both the corrected text and the preliminary audio equally to avoid modality imbalance.
  • Time-Frequency Domain Alignment: To address potential phase shifts introduced during synthesis, an alignment process is applied in the time-frequency domain.

Datasets

The paper uses the following datasets for training and evaluation:

  • Training: WHAM!, WHAMR!, and Libri2Mix. These datasets are combined for unified training.
  • Evaluation: Libri2Mix, WHAM!, and WHAMR! benchmarks are used for standard evaluation.
  • Out-of-Domain Evaluation: The test set of WSJ0-2mix is used as the speech source, with noise from the MUSAN and DEMAND datasets to simulate out-of-domain scenarios.

Key Insights and Ideas

  • Text Domain Correction is Key: Performing error correction in the text domain, rather than the audio domain, is a significant departure. The intuition is that text is a more robust and lower-dimensional representation for error correction, especially in noisy conditions.
  • Single ALM Efficiency: Using a single, powerful ALM for error correction is more efficient and less prone to cascading errors than traditional multi-stage ASR+LLM pipelines.
  • CoT for Robust Reasoning: Chain-of-Thought prompting proves effective in guiding the ALM to perform accurate transcription and correction, even in a zero-shot or few-shot setting.
  • Synthesis as Refinement: Re-synthesizing the speech based on the corrected transcription, rather than simply using the transcription as a condition for re-separation, offers a more flexible and robust approach.
  • Alignment is Crucial for Objective Metrics: While synthesis can improve naturalness, an explicit alignment step is necessary to ensure objective metrics (like SI-SNRi) are not negatively impacted by phase inconsistencies.

Key Results

SepALM consistently outperforms state-of-the-art speech separation models across various metrics, including SI-SNRi, SDRi, and subjective NMOS/SMOS scores, on the Libri2Mix, WHAM!, and WHAMR! datasets.

  • Superior Performance: SepALM achieves higher SI-SNRi and SDRi scores compared to methods like SepFormer and DiffSep. For example, on WHAMR!, SepALM (AR) achieves an SI-SNRi of 18.8 and SDRi of 19.3, outperforming SepFormer (16.4, 16.7).
  • Robustness to Out-of-Domain Noise: The text-domain correction approach demonstrates superior generalization performance on out-of-domain noise scenarios compared to methods that perform generative correction directly in the speech domain.
  • Efficiency: The single ALM approach leads to a lower Real-Time Factor (RTF) compared to cascaded methods.
  • Ablation Studies:
    • The corrector component significantly improves speech quality and reduces WER.
  • The CoT-based ALM approach rivals cascaded paradigms in performance while being more efficient.
  • The synthesis method, by treating text and audio equally, neutralizes modality imbalance and improves generalization.
  • The aligner significantly improves objective metrics like SI-SNRi.

Limitations and Future Work

  • Subjective vs. Objective Trade-off: The paper notes that the alignment phase, while improving objective metrics, has a minor detrimental effect on the naturalness (NMOS) of the separated speech. This trade-off is considered acceptable due to the overall quality improvement.
  • Future Directions: The authors highlight the potential of the speech separation-correction-synthesis-alignment paradigm for future research, particularly for even more intricate acoustic conditions. The success of ALMs in this domain suggests further exploration of their capabilities in speech processing tasks.

Summary updated

Abstract

While contemporary speech separation technologies adeptly process lengthy mixed audio waveforms, they are frequently challenged by the intricacies of real-world environments, including noisy and reverberant settings, which can result in artifacts or distortions in the separated speech. To overcome these limitations, we introduce SepALM, a pioneering approach that employs audio language models (ALMs) to rectify and re-synthesize speech within the text domain following preliminary separation. SepALM comprises four core components: a separator, a corrector, a synthesizer, and an aligner. By integrating an ALM-based end-to-end error correction mechanism, we mitigate the risk of error accumulation and circumvent the optimization hurdles typically encountered in conventional methods that amalgamate automatic speech recognition (ASR) with large language models (LLMs). Additionally, we have developed Chain-of-Thought (CoT) prompting and knowledge distillation techniques to facilitate the reasoning and training processes of the ALM. Our experiments substantiate that SepALM not only elevates the precision of speech separation but also markedly bolsters adaptability in novel acoustic environments.
BibTeX
@inproceedings{mu2025sepalm,
  author = {Zhaoxi Mu and Xinyu Yang and Gang Wang},
  title = {{SepALM: Audio Language Models Are Error Correctors for Robust Speech Separation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/912},
}
Figure