Multi-Class-Token Transformer for Multitask Self-supervised Music Information Retrieval

Yuexuan Kong, Vincent Lostanlen, Romain Hennequin, Mathieu Lagrange, Gabriel Meseguer-Brocal

Abstract

Self-supervised learning (SSL) for music information retrieval (MIR) often faces a trade-off between contrastive learning, which excels at identifying timbral and categorical features, and equivariant learning, which captures structured relationships like pitch and tonality. This paper presents the Multi-Class-Token Transformer (MT²), a compact 5.3M parameter model that bridges this gap. By employing a Vision Transformer (ViT) architecture with two specialized class tokens—one for contrastive learning and one for equivariant learning—MT² achieves state-of-the-art results across diverse MIR tasks. Despite being 18 times smaller than models like MERT, MT² demonstrates superior performance in key estimation, instrument recognition, and music tagging.

Introduction

In the field of music audio processing, self-supervised learning has emerged as a powerful paradigm for learning representations without manual labels. However, existing SSL methods typically fall into two distinct camps. Contrastive learning methods are effective at capturing "global" characteristics such as genre or instrumentation by learning representations that are invariant to transformations. Conversely, equivariant learning focuses on "structured" tasks like melody or key estimation by ensuring the representation changes predictably with specific transformations (e.g., pitch shifting).

While large-scale models have attempted to unify these objectives using Masked Language Modeling (MLM), they often require massive computational resources and parameter counts. The Multi-Class-Token Transformer (MT²) addresses this challenge by asking whether a small, efficient model can simultaneously handle both types of information without mutual interference.

The MT² Architecture

Backbone and Input Features

MT² utilizes a Vision Transformer (ViT-1D) backbone designed to process 1D patches of audio data. A key innovation in the input stage is the combination of two complementary feature sets: Mel-spectrograms, which are highly effective for capturing timbral information, and Constant-Q Transforms (CQT), which are better suited for harmonic and tonal analysis. These features are converted into patch sequences and summed before being fed into the transformer layers.

The Multi-Token Strategy

To resolve the conflict between invariance (contrastive) and sensitivity (equivariant), MT² replaces the standard single [CLS] token with two distinct class tokens:

  1. Contrastive Token ($c_{cont}$): This token is optimized using the NT-Xent loss. It learns to distinguish between different tracks, effectively clustering audio by its broad identity (e.g., instrument types or genre).
  2. Equivariant Token ($c_{equiv}$): This token is optimized using a Cross-Power Spectral Density (CPSD) loss. This objective forces the embedding space to organize itself according to musical transformations. Specifically, it encourages the model to recognize pitch shifts, leading to a representation that naturally aligns with the musical "Circle of Fifths."

Both tokens pass through the same shared transformer layers, allowing the model to learn a unified set of features while the tokens themselves specialize in different aspects of the musical signal.

Experimental Results

Pre-training and Downstream Tasks

The model was pre-trained on a private dataset of 100,000 songs. Its performance was evaluated across a wide array of downstream tasks, categorized into global tasks (tagging, instrument recognition, key estimation) and local tasks (beat tracking, chord estimation).

Performance Comparison

MT² demonstrates remarkable efficiency and efficacy. In head-to-head comparisons with MERT (95M parameters), the 5.3M parameter MT² model achieved significant gains:

  • Key Estimation: MT² achieved a MIREX score of 0.713, vastly outperforming MERT’s 0.527.
  • Instrument Recognition: MT² reached 91.8% accuracy, compared to 49.3% for MERT.
  • Music Tagging: MT² achieved 0.388 mAP on MagnaTagATune, surpassing MERT’s 0.345 mAP.

In local tasks, MT² also showed strength in chord estimation (0.447 accuracy vs. MERT's 0.392). However, it trailed slightly in beat tracking, suggesting that while the class tokens are highly specialized, the sequence tokens used for local tasks may require further optimization.

Key Insights

Specialization without Interference

A primary finding of this research is that the multi-token approach prevents "representation collapse." In single-token models, forcing a single vector to be both invariant to pitch (for tagging) and sensitive to pitch (for key detection) often leads to sub-optimal performance. MT² avoids this by allowing each token to specialize. Interestingly, the authors found that averaging the two tokens for downstream tasks often yielded the best results, indicating that the "what" (timbre) and the "how" (structure) of music are complementary.

Emergent Musical Structure

Visualization of the equivariant token's embedding space via PCA revealed a near-perfect ring structure that aligns with the musical circle of fifths. This suggests that the CPSD loss successfully encoded fundamental music theory relationships into the model's latent space without explicit supervision.

Conclusion and Future Work

MT² proves that massive parameter counts are not a prerequisite for state-of-the-art performance in music SSL. By intelligently separating contrastive and equivariant objectives through a multi-token strategy, the model achieves high accuracy and efficiency. Future work will explore adding a third token specialized for temporal and rhythmic transformations to improve performance in beat and tempo tracking, as well as scaling the approach to even larger datasets.

Summary updated

BibTeX
@inproceedings{kong2025multi,
  author = {Yuexuan Kong and Vincent Lostanlen and Romain Hennequin and Mathieu Lagrange and Gabriel Meseguer-Brocal},
  title = {{Multi-Class-Token Transformer for Multitask Self-supervised Music Information Retrieval}},
  booktitle = {WASPAA},
  year = {2025},
  doi = {10.1109/waspaa66052.2025.11231014},
}
Figure