Music Auto-Tagging with Robust Music Representation Learned via Domain Adversarial Training

Haesun Joung, Kyogu Lee

TL;DR

Existing music auto-tagging models often fail when music is layered with real-world noise, like crowd sounds or speech in a YouTube video. This paper introduces a three-step training framework using Domain Adversarial Training (DAT) to force models to learn "noise-invariant" music representations, significantly improving tagging accuracy in noisy environments even when using unlabeled data.

What Problem Does This Paper Address?

Music auto-tagging—the automated assignment of labels like "jazz," "happy," or "guitar"—works remarkably well for clean studio recordings (think Spotify). However, the performance of these models plummets when applied to "in-the-wild" audio, such as music in social media clips or videos where background noise, applause, or speech overlaps with the track.

The core challenge is that standard feature extractors produce very different embeddings for a clean track versus its noisy counterpart. The authors aim to bridge this "domain gap" between clean and noisy audio so that a classifier can identify musical traits regardless of the acoustic environment.

Methods

The authors propose a specialized architecture and a novel three-step training process based on Domain Adversarial Training (DAT).

The Architecture

The model consists of three parts:

  1. Feature Extractor (FE): Based on the SampleCNN backbone and CLMR (Contrastive Learning of Musical Representations).
  2. Domain Classifier (DC): A simple network that tries to guess if an embedding came from a "clean" or "noisy" source.
  3. Label Predictor (LP): The final stage that predicts the actual music tags (genre, mood, etc.).

The Three-Step Training Strategy

Unlike standard DAT, which often trains components simultaneously, the authors found that a staged approach prevents performance degradation:

  • Step 1: FE Pretraining. The extractor is trained using contrastive loss to learn general musical features from both clean and synthesized noisy audio.
  • Step 2: DC Pretraining. The Feature Extractor is frozen, and the Domain Classifier is trained to distinguish between clean and noisy embeddings.
  • Step 3: Adversarial Finetuning. This is the "tug-of-war." The FE is finetuned to predict tags (via the LP) while simultaneously trying to fool the DC. By using a gradient reversal layer, the FE is forced to strip away domain-specific information (the noise), resulting in embeddings that look the same to the DC whether the input is clean or noisy.

Datasets

The study utilizes several prominent audio datasets:

  • MTG-Jamendo: Used for pretraining the Feature Extractor.
  • MagnaTagATune (MTAT): The primary dataset for the auto-tagging downstream task.
  • Audioset: Used as a source of real-world non-music noise (filtered to exclude musical sounds).
  • Musan: A dataset of music, speech, and noise used to create additional "unlabeled" noisy training data and for robust testing.

Key Insights and Ideas

  • Adversarial Robustness: The "aha moment" is applying DAT—usually used for domain adaptation in images or speech—to the specific problem of music noise. By making the model "blind" to the presence of noise, it becomes better at "hearing" the underlying music.
  • The Power of Unlabeled Data: One of the most significant findings is that the model can improve by using unlabeled noisy data. In their "Proposed (b)" configuration, adding noisy samples without any ground-truth tags still helped the Feature Extractor learn more robust representations.
  • Staged Training Matters: The authors observed that training the Domain Classifier alongside other components from the start led to lower performance. Pretraining the DC separately (Step 2) proved crucial for stable adversarial learning in Step 3.

Key Results

The researchers evaluated their model using Area Under the ROC Curve (AUC) and Average Precision (AP) across various Signal-to-Noise Ratios (SNR).

  • Superiority in Noise: In high-noise scenarios (SNR -5), the proposed adversarial models consistently outperformed the baseline. For example, on the Audioset noise test, the proposed model achieved an AUC of 0.604, compared to the baseline's 0.595.
  • Generalization: As the variety of noise types increased (from 1 noise type to 4), the proposed method's performance gap over the baseline widened, suggesting better generalization to complex, real-world environments.
  • Efficiency: The "Proposed (b)" model, which used a smaller but more diverse set of unlabeled noisy data, often outperformed "Proposed (a)," which used a larger but less diverse set. This suggests that noise diversity is more important than raw data volume for building robust music models.

Limitations and Future Work

  • Synthesized vs. Real Noise: While the authors used real-world noise samples (Audioset/Musan), they were mixed with clean music programmatically. Testing on "true" in-the-wild recordings where music and noise are captured by the same microphone (including reverb and distance effects) remains a future step.
  • Computational Overhead: The three-step training process adds complexity compared to standard supervised learning.
  • Tag Vocabulary: The study focuses on the 50 multi-tags of the MTAT dataset; expanding this to a broader or more hierarchical taxonomy of musical tags would be a logical next step.

Summary updated

Abstract

Music auto-tagging is crucial for enhancing music discovery and recommendation. Existing models in Music Information Retrieval (MIR) struggle with real-world noise such as environmental and speech sounds in multimedia content. This study proposes a method inspired by speech-related tasks to enhance music auto-tagging performance in noisy settings. The approach integrates Domain Adversarial Training (DAT) into the music domain, enabling robust music representations that withstand noise. Unlike previous research, this approach involves an additional pretraining phase for the domain classifier, to avoid performance degradation in the subsequent phase. Adding various synthesized noisy music data improves the model’s generalization across different noise levels. The proposed architecture demonstrates enhanced performance in music auto-tagging by effectively utilizing unlabeled noisy music data. Additional experiments with supplementary unlabeled data further improves the model’s performance, underscoring its robust generalization capabilities and broad applicability.
BibTeX
@inproceedings{joung2024music,
  author = {Haesun Joung and Kyogu Lee},
  title = {{Music Auto-Tagging with Robust Music Representation Learned via Domain Adversarial Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=MDJolIZFEP},
}

Loading related content...

Figure