Abstract
Self-supervised music representation models typically rely on contrastive learning, maximizing similarity between distorted versions of the same track. However, these methods often treat all audio frames as equally significant, ignoring the inherent variability in music, such as silence or background noise. We present PEMR (Positive-nEgative frame mask for Music Representation), a framework that identifies "crucial" versus "inessential" frames using a Transformer-based attention mechanism. By generating augmented positive samples and "counterfactual" negative samples from the same track, PEMR learns more robust representations, achieving state-of-the-art results in music classification and cover song identification.
The Challenge of Coarse-Grained Contrastive Learning
Most existing self-supervised models for music representation utilize contrastive learning: they take two augmented versions of a song and force their latent representations to be similar. While effective, this "coarse-grained" approach assumes all time-slices (frames) of an audio segment are equally representative of the song's identity.
In practice, music is heterogeneous. A clip may contain silence, non-musical noise, or generic transitions that do not capture the core melodic or harmonic essence of the track. Forcing a model to align a noisy or silent frame with a highly melodic one can lead to "blurry" representations. The PEMR framework addresses this by distinguishing between crucial and inessential frames during training.
The PEMR Framework
The authors propose a two-stage approach to masking that leverages the internal attention of a Transformer model to guide the contrastive process.
1. The Predicting Module
Before masking can occur, the model must understand the contextual relationships between frames. A Transformer encoder is trained using a random masking strategy—similar to BERT in natural language processing—on Log-Mel spectrograms. This allows the model to learn how different parts of a song correlate with one another.
2. Positive-Negative Mask Generation
This is the core innovation of PEMR. Using the attention scores from a [CLS] token, the model ranks the importance of each frame:
- Positive Mask: The model masks "inessential" (low-attention) frames. The remaining "crucial" parts form an augmented positive sample that retains the song's core identity.
- Negative Mask: The model masks "crucial" (high-attention) frames. This creates a "counterfactual" negative sample—a version of the song that lacks its defining characteristics, even though it originates from the same track.
3. Contrastive Objective
The framework utilizes a multi-part loss function, including Huber loss for reconstruction and a modified Barlow Twins loss for the contrastive task. This setup prevents "model collapse," where the encoder might otherwise output trivial, constant values.
Key Insights: Intra-Song Negatives and Attention Filters
PEMR shifts the paradigm of negative sampling. Traditionally, negative samples are drawn from different songs. By generating negative samples from the same song (by removing its most important parts), PEMR forces the encoder to learn exactly which features define a specific track. Furthermore, instead of relying on stochastic data augmentations like pitch shifting or noise injection, PEMR uses the model's own "understanding" of the audio to decide what to emphasize.
Experimental Results
The researchers evaluated PEMR across several benchmark datasets, including MagnaTagATune (MTAT) for classification and SHS100K for cover song identification (CSI).
- State-of-the-Art Performance: On the MTAT dataset, PEMR achieved a 90.3% ROC-AUC, outperforming established self-supervised methods such as SimCLR and MoCo.
- Data Efficiency: In semi-supervised settings with only 1% labeled data, PEMR reached a 77.3% ROC-AUC, significantly higher than a standard supervised FCN (73.2%).
- Transferability: When pre-trained on GTZAN and tested on MTAT, the model maintained a high 90.0% ROC-AUC, demonstrating that the learned features generalize well across different musical contexts.
- Cover Song Identification: Integrating PEMR’s pre-trained encoder into supervised CSI models like CQT-Net resulted in a Mean Average Precision (MAP) jump from 0.446 to 0.484 on the SHS100K-SUB test set.
Sensitivity and Limitations
A critical hyperparameter in PEMR is the masking ratio (τ). The effectiveness of the positive and negative samples depends heavily on this balance.
As shown in the figure above, performance peaks at a ratio of approximately 0.1. If the ratio is too high (e.g., 0.5), the negative samples may still contain too much crucial information, while the positive samples become too sparse, leading to a noticeable decline in accuracy. Additionally, the Transformer-based architecture introduces higher computational overhead compared to simpler CNN-based methods. Future work may explore applying these representations to broader Music Information Retrieval (MIR) tasks such as music recommendation and source separation.
Summary updated