TL;DR
This paper tackles the growing problem of copyright infringement on short video platforms, where users obscure original audio with background music to bypass detection. They introduce a novel deep learning pipeline that effectively separates and removes this background music, restoring the original audio. To support this, they also release two new datasets, OASD-20K and OSVAR-160, specifically designed for this audio restoration task.
What Problem Does This Paper Address?
Short video platforms like TikTok and YouTube Shorts have exploded in popularity, but this user-generated content comes with a significant copyright challenge. A common tactic to evade detection systems is to overlay arbitrary background music (BGM) onto original videos, masking the original soundtrack (OST) or dialogue. This not only complicates copyright compliance but also hinders the effectiveness of audio-based retrieval systems used to identify duplicate or infringing content. Existing methods struggle to accurately separate and restore the original audio in these complex, blended scenarios, necessitating a more robust solution.
Methods
The core of this work is a four-module deep learning pipeline designed for audio restoration in short videos:
- Background Music Separator & Speech Extractor: This initial stage uses the Demucs model for general audio decomposition and Pyannote Audio VAD to isolate speech. The goal is to create a "dialogue-only video" by separating speech from the mixed audio.
- Mixed-Music Separator: This module employs the BS-RoFormer model, fine-tuned on their new OASD-20K dataset. BS-RoFormer is designed to handle complex audio mixtures by processing signals across different frequency bands and using positional embeddings to capture temporal and frequency dependencies, making it suitable for separating OST from BGM.
- Video-Music Matching Classifier: This module leverages a fine-tuned UT-CMVMR model. It uses Conformer-based modules to extract video and audio embeddings, incorporating optical flow and rhythm quantization features. The model is trained using Triplet Loss to learn the similarity between video content and music, aiming to recommend appropriate music or identify the original OST.
The pipeline is designed to sequentially process the audio, first separating BGM and then restoring the OST, ultimately aiming to preserve content integrity.
Datasets
The paper introduces two novel datasets:
- OASD-20K (Oh-Audio Separation Dataset 20K): This dataset contains 20,000 four-second audio clips created by mixing OST tracks from Korean TV series with BGM from the YouTube Audio Library. It's designed for training and evaluating mixed-music separation models.
- OSVAR-160 (Oh-Short Video Audio Restoration Dataset 160): This benchmark dataset comprises 1,121 one-minute video clips from Korean TV series, each paired with an arbitrarily overlaid BGM track. It's specifically for evaluating the end-to-end performance of the entire pipeline in realistic short video scenarios.
They also adapted a subset of the HIMV-200K dataset for fine-tuning the Video-Music Matching module.
Key Insights and Ideas
- Domain-Specific Fine-Tuning is Crucial: The paper strongly emphasizes that fine-tuning existing powerful models (like BS-RoFormer and UT-CMVMR) on datasets specifically curated for short video content (OASD-20K and OSVAR-160) significantly improves performance.
- BGM Obscures More Than Just Copyright: The overlaying of BGM doesn't just bypass copyright checks; it actively degrades the structural quality of the original audio, as evidenced by the SI-SDR metric.
- A Multi-Stage Pipeline is Necessary: No single model can perfectly handle both BGM removal and OST restoration. The proposed pipeline's modular approach, with distinct stages for initial separation, mixed-music separation, and video-music alignment, is key to its success.
- SI-SDR is a Better Indicator of Structural Restoration: While SDR measures overall signal quality, SI-SDR is highlighted as a more critical metric for assessing how well the original audio's structure is preserved after BGM removal, especially in scenarios with dynamic volume variations.
Key Results
- Arbitrary BGM Removal: The pipeline achieves impressive performance in arbitrary BGM separation, with an average SDR of 138.63 dB and SI-SDR of 16.53 dB.
- OST Restoration: While OST separation scores are slightly lower (SDR of 8.23 dB, SI-SDR of 7.53 dB), the results are still robust and significantly better than scenarios where BGM interference is present.
- Video-Music Matching Accuracy: The fine-tuned UT-CMVMR model achieves a high accuracy of 96.8% in matching videos with their intended music, outperforming previous benchmarks.
- Inference Time: The entire pipeline can process a video in approximately 9.17 seconds, making it potentially viable for real-time applications.
Limitations and Future Work
- Preservation of Non-Musical Audio: The paper acknowledges that preserving non-musical audio effects (like ambient sounds or sound effects) alongside dialogue and OST is an area for future improvement.
- Dataset Expansion: Expanding the datasets to include a wider variety of genres and languages is suggested to enhance the model's adaptability to global platforms.
- Further Quality Improvements: The authors aim to further improve the quality of separated audio components.
Summary updated