Abstract
In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.
Introduction
The "cocktail party problem"—the challenge of isolating a single voice from a noisy environment—has long been a focal point of audio research. However, the rise of short-form video platforms like TikTok and live streaming services has introduced a more complex variation: the simultaneous overlap of speech, singing voices, and background music (BGM).
Standard Automatic Speech Recognition (ASR) systems are typically optimized for clean speech. When these systems encounter singing or heavy BGM, their performance degrades significantly. To address this, researchers from the Chinese Academy of Sciences and Tencent AI Lab have developed JRSV (Jointly Recognize Speech and singing Voices), a framework designed to disentangle and transcribe these overlapping audio streams.
The JRSV Framework
The JRSV architecture addresses the limitations of traditional cascade systems (which separate then recognize) and end-to-end multi-talker models (which often struggle with track identity). It consists of two primary components:
- Multi-Task Audio Source Separation (MTASS): This Conformer-based network processes mixed audio to output two distinct spectral magnitudes: one for speech and one for singing. Unlike Permutation Invariant Training (PIT) models that may swap tracks, MTASS uses task-specific output layers. This ensures the model consistently identifies which track is speech and which is singing.
- ASR Module: Utilizing a "U2" structure (CTC/Attention rescoring), this module performs parallel recognition on both separated tracks, converting the isolated signals into text.
Overcoming Training Challenges
The researchers identified two major hurdles in training such a complex system: the "joint training paradox" and separation artifacts.
Two-Stage Curriculum Learning
Initial attempts to optimize the separation and recognition modules simultaneously led to poor convergence. The authors discovered that low-level signal separation and high-level semantic recognition are fundamentally different tasks that can interfere with each other during early training. To solve this, they implemented a two-stage approach: first training the MTASS module independently, then freezing it while training the ASR module. This curriculum ensures the ASR receives stable, albeit imperfect, inputs from the start.
Online Distillation
Audio separation often introduces "artifacts"—unnatural distortions that can confuse a standard ASR. To bridge the gap between clean and separated audio, JRSV employs online distillation. The ASR encoder is trained to produce internal representations for the separated audio that match the representations of the original clean audio. This teaches the model to focus on semantic content while ignoring the acoustic imperfections introduced by the separation process.
Experimental Setup and Dataset
To evaluate the model, the team created the Dual-Track Speech and singing Voice Dataset (DTSVD). This benchmark was constructed by mixing three distinct sources:
- AISHELL-1: Clean Chinese speech.
- OpenSinger: Chinese singing voices.
- MusDB18: Background music.
The data was mixed dynamically during training with varying overlap ratios (from 0 to 1.0) and different signal-to-noise ratios (SNRs) to simulate the unpredictable nature of real-world social media audio.
Results and Discussion
Experimental results indicate that JRSV significantly outperforms traditional methods. The version of the model utilizing frozen MTASS and distillation (JRSV-f-d) achieved:
- 12.3% Character Error Rate (CER) for speech.
- 12.1% CER for singing voices.
Compared to the best-performing cascade systems, JRSV reduced errors by 41% for speech and 57% for singing. Notably, the model remained robust even in scenarios with a 1.0 overlap ratio (where speech and singing occur simultaneously throughout), a condition where standard ASR models typically fail due to massive insertion errors.
Conclusion
JRSV represents a significant step forward in processing the complex audio environments of modern digital media. By combining structured source separation with robust distillation techniques, the framework successfully navigates the interference between speech, song, and music. Future work will focus on refining joint optimization strategies and expanding the model's robustness to include more diverse environmental noises, such as wind or street sounds.
Summary updated