Contrastive Learning with Positive-Negative Frame Mask for Music Representation

Dong Yao, Zhou Zhao, Shengyu Zhang, Jieming Zhu, Yudong Zhu, Rui Zhang, Xiuqiang He

Abstract

Self-supervised music representation models typically rely on contrastive learning, maximizing similarity between distorted versions of the same track. However, these methods often treat all audio frames as equally significant, ignoring the inherent variability in music, such as silence or background noise. We present PEMR (Positive-nEgative frame mask for Music Representation), a framework that identifies "crucial" versus "inessential" frames using a Transformer-based attention mechanism. By generating augmented positive samples and "counterfactual" negative samples from the same track, PEMR learns more robust representations, achieving state-of-the-art results in music classification and cover song identification.

The Challenge of Coarse-Grained Contrastive Learning

Most existing self-supervised models for music representation utilize contrastive learning: they take two augmented versions of a song and force their latent representations to be similar. While effective, this "coarse-grained" approach assumes all time-slices (frames) of an audio segment are equally representative of the song's identity.

In practice, music is heterogeneous. A clip may contain silence, non-musical noise, or generic transitions that do not capture the core melodic or harmonic essence of the track. Forcing a model to align a noisy or silent frame with a highly melodic one can lead to "blurry" representations. The PEMR framework addresses this by distinguishing between crucial and inessential frames during training.

The PEMR Framework

The authors propose a two-stage approach to masking that leverages the internal attention of a Transformer model to guide the contrastive process.

The PEMR architecture, featuring the Predicting Module and the Positive-Negative Mask Generation Module.

1. The Predicting Module

Before masking can occur, the model must understand the contextual relationships between frames. A Transformer encoder is trained using a random masking strategy—similar to BERT in natural language processing—on Log-Mel spectrograms. This allows the model to learn how different parts of a song correlate with one another.

2. Positive-Negative Mask Generation

This is the core innovation of PEMR. Using the attention scores from a [CLS] token, the model ranks the importance of each frame:

  • Positive Mask: The model masks "inessential" (low-attention) frames. The remaining "crucial" parts form an augmented positive sample that retains the song's core identity.
  • Negative Mask: The model masks "crucial" (high-attention) frames. This creates a "counterfactual" negative sample—a version of the song that lacks its defining characteristics, even though it originates from the same track.

3. Contrastive Objective

The framework utilizes a multi-part loss function, including Huber loss for reconstruction and a modified Barlow Twins loss for the contrastive task. This setup prevents "model collapse," where the encoder might otherwise output trivial, constant values.

Key Insights: Intra-Song Negatives and Attention Filters

PEMR shifts the paradigm of negative sampling. Traditionally, negative samples are drawn from different songs. By generating negative samples from the same song (by removing its most important parts), PEMR forces the encoder to learn exactly which features define a specific track. Furthermore, instead of relying on stochastic data augmentations like pitch shifting or noise injection, PEMR uses the model's own "understanding" of the audio to decide what to emphasize.

Experimental Results

The researchers evaluated PEMR across several benchmark datasets, including MagnaTagATune (MTAT) for classification and SHS100K for cover song identification (CSI).

  • State-of-the-Art Performance: On the MTAT dataset, PEMR achieved a 90.3% ROC-AUC, outperforming established self-supervised methods such as SimCLR and MoCo.
  • Data Efficiency: In semi-supervised settings with only 1% labeled data, PEMR reached a 77.3% ROC-AUC, significantly higher than a standard supervised FCN (73.2%).
  • Transferability: When pre-trained on GTZAN and tested on MTAT, the model maintained a high 90.0% ROC-AUC, demonstrating that the learned features generalize well across different musical contexts.
  • Cover Song Identification: Integrating PEMR’s pre-trained encoder into supervised CSI models like CQT-Net resulted in a Mean Average Precision (MAP) jump from 0.446 to 0.484 on the SHS100K-SUB test set.

Sensitivity and Limitations

A critical hyperparameter in PEMR is the masking ratio (τ). The effectiveness of the positive and negative samples depends heavily on this balance.

Impact of the masking ratio on model performance (ROC-AUC and PR-AUC).

As shown in the figure above, performance peaks at a ratio of approximately 0.1. If the ratio is too high (e.g., 0.5), the negative samples may still contain too much crucial information, while the positive samples become too sparse, leading to a noticeable decline in accuracy. Additionally, the Transformer-based architecture introduces higher computational overhead compared to simpler CNN-based methods. Future work may explore applying these representations to broader Music Information Retrieval (MIR) tasks such as music recommendation and source separation.

Summary updated

Abstract

Self-supervised learning, especially contrastive learning, has made an outstanding contribution to the development of many deep learning research fields. Recently, researchers in the acoustic signal processing field noticed its success and leveraged contrastive learning for better music representation. Typically, existing approaches maximize the similarity between two distorted audio segments sampled from the same music. In other words, they ensure a semantic agreement at the music level. However, those coarse-grained methods neglect some inessential or noisy elements at the frame level, which may be detrimental to the model to learn the effective representation of music. Towards this end, this paper proposes a novel Positive-nEgative frame mask for Music Representation based on the contrastive learning framework, abbreviated as PEMR. Concretely, PEMR incorporates a Positive-Negative Mask Generation module, which leverages transformer blocks to generate frame masks on the Log-Mel spectrogram. We can generate self-augmented negative and positive samples by masking important components or inessential components, respectively. We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music. We conduct experiments on four public datasets. The experimental results of two music-related downstream tasks, music classification, and cover song identification, demonstrate the generalization ability and transferability of music representation learned by PEMR.
BibTeX
@inproceedings{yao2022contrastive,
  author = {Dong Yao and Zhou Zhao and Shengyu Zhang and Jieming Zhu and Yudong Zhu and Rui Zhang and Xiuqiang He},
  title = {{Contrastive Learning with Positive-Negative Frame Mask for Music Representation}},
  booktitle = {The Web Conference},
  year = {2022},
  doi = {10.1145/3485447.3512011},
}
Figure