Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

Ye Bai, Chenxing Li, Hao Li, Yuanyuan Zhao, Xiaorui Wang

This paper, recently published by researchers from the Chinese Academy of Sciences and Tencent AI Lab, tackles the "cocktail party problem" with a modern twist: how do we recognize and transcribe audio when someone is talking over a singing voice, all while background music is playing?

TL;DR

The authors propose JRSV (Jointly Recognize Speech and singing Voices), a framework that combines multi-task audio source separation with automatic speech recognition (ASR). By using a two-stage training process and online distillation, the model achieves a 41% reduction in character error rates (CER) for speech and a 57% reduction for singing voices compared to traditional cascade systems.

What Problem Does This Paper Address?

In the world of short-form video (TikTok, Reels) and live streaming, audio is rarely "clean." Speech, singing, and background music (BGM) frequently overlap. Standard ASR systems are designed for clean speech; when faced with a singing voice or heavy BGM, their performance plummets.

Existing solutions usually fall into two camps:

  1. Cascade Systems: Separate the audio first, then run ASR. The problem? Separation often introduces "artifacts" (distortions) that confuse the ASR.
  2. End-to-End Multi-talker ASR: These can handle multiple speakers but often suffer from the "permutation problem"—the model doesn't know which output track is the speaker and which is the singer.

JRSV aims to solve both by creating a system that knows exactly what it is separating and uses specialized training to stay robust against separation errors.

Methods

The JRSV architecture consists of two primary components:

  • MTASS (Multi-Task Audio Source Separation): A Conformer-based network that takes mixed audio and outputs two distinct spectral magnitudes: one for speech and one for singing. Crucially, it is "PIT-free" (Permutation Invariant Training-free) because the output layers are hard-coded to specific tasks, ensuring the model always knows which track is which.
  • ASR Module: A "U2" structure (CTC/Attention rescoring) that recognizes the content of both tracks in parallel.

The "Secret Sauce"

The researchers introduced two critical training techniques:

  1. Two-Stage Curriculum Learning: They found that training the whole system at once led to poor convergence. Instead, they first trained the MTASS module, then froze it while training the ASR module. They argue that low-level signal separation and high-level semantic recognition are fundamentally different tasks that can interfere with each other if optimized simultaneously from scratch.
  2. Online Distillation: To bridge the gap between "perfect" clean audio and "imperfect" separated audio, they used distillation. They forced the ASR encoder to produce similar internal representations for the separated audio as it does for the original clean audio, effectively teaching the model to ignore separation artifacts.

Datasets

The authors introduced a new benchmark: DTSVD (Dual-Track Speech and singing Voice Dataset). It is built by mixing three existing datasets:

  • AISHELL-1: For clean Chinese speech.
  • OpenSinger: For Chinese singing voices.
  • MusDB18: For background music.

The data was mixed "on-the-fly" during training with varying overlap ratios (from no overlap to total overlap) and different signal-to-noise ratios (SNRs) to ensure the model could handle diverse real-world scenarios.

Key Insights and Ideas

  • Separation as a Pre-conditioner: Unlike previous end-to-end models that try to do everything at once, this paper argues for a structured separation step that preserves the "identity" of the audio source (speech vs. lyric).
  • The Joint Training Paradox: One of the most interesting findings was that joint optimization of separation and recognition losses actually hurt performance. The authors suggest that the ASR's high-level semantic requirements are "incompatible" with the low-level acoustic requirements of the separation module during early training.
  • Distillation is Essential: Without online distillation, the ASR module struggles with the "unnatural" sounds produced by the separation module. Distillation acts as a robust feature extractor that focuses on the content rather than the signal quality.

Key Results

The JRSV model was tested against several baselines, including ASR models trained on mixed data and traditional cascade systems.

  • Accuracy: JRSV-f-d (the version with frozen MTASS and distillation) achieved an average 12.3% CER for speech and 12.1% CER for singing.
  • Relative Improvement: Compared to the best cascade system, JRSV reduced errors by 41% for speech and 57% for singing.
  • Robustness: Even at 1.0 overlap (where speech and singing happen simultaneously for the entire duration), the model maintained high accuracy, whereas standard ASR models failed completely (often exceeding 100% CER due to massive insertion errors).

Limitations and Future Work

The authors acknowledge that while their two-stage training works, they haven't yet cracked the code for successful joint optimization. They plan to investigate why combining the losses ($L_{MTASS}$ and $L_{ASR}$) doesn't currently yield better results. Additionally, while the dataset covers speech, singing, and music, real-world environments contain even more complex noise (street noise, wind, etc.) that could be integrated into future iterations.

Summary updated

Abstract

In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.
BibTeX
@inproceedings{bai2024jointly,
  author = {Ye Bai and Chenxing Li and Hao Li and Yuanyuan Zhao and Xiaorui Wang},
  title = {{Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=keOwNY9g8L},
}

Loading related content...

Figure