TL;DR
Existing music auto-tagging models often fail when music is layered with real-world noise, like crowd sounds or speech in a YouTube video. This paper introduces a three-step training framework using Domain Adversarial Training (DAT) to force models to learn "noise-invariant" music representations, significantly improving tagging accuracy in noisy environments even when using unlabeled data.
What Problem Does This Paper Address?
Music auto-tagging—the automated assignment of labels like "jazz," "happy," or "guitar"—works remarkably well for clean studio recordings (think Spotify). However, the performance of these models plummets when applied to "in-the-wild" audio, such as music in social media clips or videos where background noise, applause, or speech overlaps with the track.
The core challenge is that standard feature extractors produce very different embeddings for a clean track versus its noisy counterpart. The authors aim to bridge this "domain gap" between clean and noisy audio so that a classifier can identify musical traits regardless of the acoustic environment.
Methods
The authors propose a specialized architecture and a novel three-step training process based on Domain Adversarial Training (DAT).
The Architecture
The model consists of three parts:
- Feature Extractor (FE): Based on the SampleCNN backbone and CLMR (Contrastive Learning of Musical Representations).
- Domain Classifier (DC): A simple network that tries to guess if an embedding came from a "clean" or "noisy" source.
- Label Predictor (LP): The final stage that predicts the actual music tags (genre, mood, etc.).
The Three-Step Training Strategy
Unlike standard DAT, which often trains components simultaneously, the authors found that a staged approach prevents performance degradation:
- Step 1: FE Pretraining. The extractor is trained using contrastive loss to learn general musical features from both clean and synthesized noisy audio.
- Step 2: DC Pretraining. The Feature Extractor is frozen, and the Domain Classifier is trained to distinguish between clean and noisy embeddings.
- Step 3: Adversarial Finetuning. This is the "tug-of-war." The FE is finetuned to predict tags (via the LP) while simultaneously trying to fool the DC. By using a gradient reversal layer, the FE is forced to strip away domain-specific information (the noise), resulting in embeddings that look the same to the DC whether the input is clean or noisy.
Datasets
The study utilizes several prominent audio datasets:
- MTG-Jamendo: Used for pretraining the Feature Extractor.
- MagnaTagATune (MTAT): The primary dataset for the auto-tagging downstream task.
- Audioset: Used as a source of real-world non-music noise (filtered to exclude musical sounds).
- Musan: A dataset of music, speech, and noise used to create additional "unlabeled" noisy training data and for robust testing.
Key Insights and Ideas
- Adversarial Robustness: The "aha moment" is applying DAT—usually used for domain adaptation in images or speech—to the specific problem of music noise. By making the model "blind" to the presence of noise, it becomes better at "hearing" the underlying music.
- The Power of Unlabeled Data: One of the most significant findings is that the model can improve by using unlabeled noisy data. In their "Proposed (b)" configuration, adding noisy samples without any ground-truth tags still helped the Feature Extractor learn more robust representations.
- Staged Training Matters: The authors observed that training the Domain Classifier alongside other components from the start led to lower performance. Pretraining the DC separately (Step 2) proved crucial for stable adversarial learning in Step 3.
Key Results
The researchers evaluated their model using Area Under the ROC Curve (AUC) and Average Precision (AP) across various Signal-to-Noise Ratios (SNR).
- Superiority in Noise: In high-noise scenarios (SNR -5), the proposed adversarial models consistently outperformed the baseline. For example, on the Audioset noise test, the proposed model achieved an AUC of 0.604, compared to the baseline's 0.595.
- Generalization: As the variety of noise types increased (from 1 noise type to 4), the proposed method's performance gap over the baseline widened, suggesting better generalization to complex, real-world environments.
- Efficiency: The "Proposed (b)" model, which used a smaller but more diverse set of unlabeled noisy data, often outperformed "Proposed (a)," which used a larger but less diverse set. This suggests that noise diversity is more important than raw data volume for building robust music models.
Limitations and Future Work
- Synthesized vs. Real Noise: While the authors used real-world noise samples (Audioset/Musan), they were mixed with clean music programmatically. Testing on "true" in-the-wild recordings where music and noise are captured by the same microphone (including reverb and distance effects) remains a future step.
- Computational Overhead: The three-step training process adds complexity compared to standard supervised learning.
- Tag Vocabulary: The study focuses on the 50 multi-tags of the MTAT dataset; expanding this to a broader or more hierarchical taxonomy of musical tags would be a logical next step.
Summary updated