Contrastive Learning with Positive-Negative Frame Mask for Music Representation

Dong Yao, Zhou Zhao, Shengyu Zhang, Jieming Zhu, Yudong Zhu, Rui Zhang, Xiuqiang He

Abstract

Self-supervised learning, especially contrastive learning, has made an outstanding contribution to the development of many deep learning research fields. Recently, researchers in the acoustic signal processing field noticed its success and leveraged contrastive learning for better music representation. Typically, existing approaches maximize the similarity between two distorted audio segments sampled from the same music. In other words, they ensure a semantic agreement at the music level. However, those coarse-grained methods neglect some inessential or noisy elements at the frame level, which may be detrimental to the model to learn the effective representation of music. Towards this end, this paper proposes a novel Positive-nEgative frame mask for Music Representation based on the contrastive learning framework, abbreviated as PEMR. Concretely, PEMR incorporates a Positive-Negative Mask Generation module, which leverages transformer blocks to generate frame masks on the Log-Mel spectrogram. We can generate self-augmented negative and positive samples by masking important components or inessential components, respectively. We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music. We conduct experiments on four public datasets. The experimental results of two music-related downstream tasks, music classification, and cover song identification, demonstrate the generalization ability and transferability of music representation learned by PEMR.

Introduction

In Music Information Retrieval (MIR), the scarcity of labeled data is a significant bottleneck. While self-supervised contrastive learning has emerged as a powerful solution—teaching models to recognize similarities between distorted versions of the same track—standard methods often treat audio clips as monolithic entities. Music, however, is inherently non-uniform; a single clip may contain silence, noise, or varying levels of thematic importance across its duration.

Existing coarse-grained approaches treat every frame in a spectrogram with equal weight, potentially diluting the learned representation with irrelevant information. This paper introduces PEMR (Positive-nEgative frame mask for Music Representation), a framework designed to identify and leverage the varying importance of individual frames to create more robust music representations.

The PEMR Framework

The core innovation of PEMR lies in its move from random data augmentation to a surgical, attention-driven masking strategy at the frame level.

The PEMR architecture overview

1. Predicting Module

The process begins by training a Transformer encoder using a random masking strategy. By predicting masked frames, the model learns the contextual and temporal correlations within a Log-Mel spectrogram.

2. Positive-Negative Mask Generation

Using attention scores from the Transformer's [CLS] token, PEMR identifies "crucial" frames (high attention) and "inessential" frames (low attention):

  • Positive Mask: Erases inessential frames, forcing the model to represent the song using only its most informative components.
  • Negative Mask: Erases the crucial frames. This creates a "counterfactual" negative sample—a version of the same track that lacks its defining characteristics, providing a harder challenge for the model to distinguish during training.

3. Contrastive Objective

The framework employs a combination of Huber loss for frame prediction and Barlow Twins loss for the contrastive task. The Barlow Twins objective is particularly effective at preventing model collapse by optimizing the cross-correlation matrix of the learned features.

Experimental Evaluation

The researchers evaluated PEMR across four datasets: MagnaTagATune (MTAT) and GTZAN for classification, and SHS100K and Covers80 for cover song identification.

  • State-of-the-Art Performance: On the MTAT dataset, PEMR achieved a 90.3% ROC-AUC and 38.0% PR-AUC, outperforming previous benchmarks like CLMR and BYOL-A.
  • Cover Song Identification: Integrating the PEMR encoder into supervised models like CQT-Net resulted in a significant jump in Mean Average Precision (MAP) on the SHS100K-SUB dataset, rising from 0.446 to 0.484.
  • Data Efficiency: PEMR demonstrated remarkable robustness in low-data regimes. When trained on only 1% of labeled MTAT data, it significantly outperformed standard supervised baselines.

Sensitivity and Insights

A critical factor in PEMR's success is the masking ratio ($r$). The model's performance is sensitive to how much information is removed to create positive and negative samples.

Impact of masking ratio on performance

As shown in the sensitivity analysis, a masking ratio of 0.1 (10%) was found to be the optimal "sweet spot." If the ratio is too high, the positive samples lose too much essential information; if it is too low, the generated negative samples are not challenging enough to force the model to learn deep features.

Conclusion

PEMR advances music representation learning by shifting the focus from global audio segments to fine-grained frame importance. By generating "hard negatives" from within the same track, the model learns more robust and generalized features. Future work may extend these representations to other MIR tasks such as source separation, music recommendation, and instrument recognition.

Summary updated

Abstract

Self-supervised learning, especially contrastive learning, has made an outstanding contribution to the development of many deep learning research fields. Recently, researchers in the acoustic signal processing field noticed its success and leveraged contrastive learning for better music representation. Typically, existing approaches maximize the similarity between two distorted audio segments sampled from the same music. In other words, they ensure a semantic agreement at the music level. However, those coarse-grained methods neglect some inessential or noisy elements at the frame level, which may be detrimental to the model to learn the effective representation of music. Towards this end, this paper proposes a novel Positive-nEgative frame mask for Music Representation based on the contrastive learning framework, abbreviated as PEMR. Concretely, PEMR incorporates a Positive-Negative Mask Generation module, which leverages transformer blocks to generate frame masks on the Log-Mel spectrogram. We can generate self-augmented negative and positive samples by masking important components or inessential components, respectively. We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music. We conduct experiments on four public datasets. The experimental results of two music-related downstream tasks, music classification, and cover song identification, demonstrate the generalization ability and transferability of music representation learned by PEMR.
BibTeX
@inproceedings{yao2022contrastive,
  author = {Dong Yao and Zhou Zhao and Shengyu Zhang and Jieming Zhu and Yudong Zhu and Rui Zhang and Xiuqiang He},
  title = {{Contrastive Learning with Positive-Negative Frame Mask for Music Representation}},
  booktitle = {The Web Conference},
  year = {2022},
  doi = {10.1145/3485447.3512011},
}
Figure