Mertech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model with Multi-Task Finetuning

Dichucheng Li, Yinghao Ma, Weixing Wei, Qiuqiang Kong, Yulun Wu, Mingjin Che, Fan Xia, Emmanouil Benetos, Wei Li

TL;DR

MERTech is a new framework for detecting instrument playing techniques (IPTs)—like vibratos or glissandos—by finetuning a self-supervised model (MERT) originally trained on massive amounts of unlabeled music. By treating pitch and onset detection as "auxiliary" tasks and using a clever post-processing "gate," the researchers achieved state-of-the-art results, even on niche instruments with very little labeled data.

What Problem Does This Paper Address?

In music information retrieval (MIR), detecting Instrument Playing Techniques (IPTs) is a "boss level" challenge. While identifying a guitar or a piano is relatively straightforward for AI, identifying how it's being played (e.g., a "hammer-on" vs. a "pull-off") is much harder.

The field faces two major hurdles:

  1. Data Scarcity: Labeling IPTs requires expert-level musical knowledge and is incredibly time-consuming. Most datasets only have a couple of hours of audio.
  2. Class Imbalance: In a typical song, most notes are played "normally." Specific techniques are rare embellishments, making it hard for standard supervised models to learn the "minority" classes without overfitting.

Methods

The authors introduce MERTech, which moves away from training models from scratch and instead leverages Transfer Learning.

  • The Backbone: They use MERT-v1-95M, a self-supervised model pre-trained on 160,000 hours of unlabeled music. MERT is designed to understand musical "language" much like BERT understands human text.
  • Multi-Task Finetuning: Instead of just asking the model "What technique is this?", they train it to simultaneously predict pitch and IPT onsets. The logic is intuitive: a glissando is defined by its pitch movement, and knowing exactly when a technique starts (onset) helps the model locate the event in time.
  • Architecture: The model takes a weighted sum of MERT’s internal layers (capturing different levels of musical abstraction) and feeds them into two branches. These branches use Multi-Layer Perceptrons (MLPs) and self-attention refinement layers to produce the final predictions.
  • Post-Processing "Gate": To improve event-level accuracy, they implemented a rule: an IPT activation only counts as a new "event" if the model’s onset detector also triggers a "start" signal in that same frame.

Datasets

The researchers evaluated MERTech on three distinct benchmarks to prove its versatility:

  • Guzheng_Tech99: 99 polyphonic pieces for the Guzheng (a Chinese zither), featuring 7 techniques like vibrato and tremolo.
  • EG-Solo: 76 electric guitar solos with 9 techniques, including bends, slides, and harmonics.
  • CBFdataset: 80 monophonic Chinese bamboo flute performances with 7 techniques.

Notably, while MERT was pre-trained mostly on Western music, this paper tests how well that knowledge transfers to traditional Chinese instruments.

Key Insights and Ideas

  • SSL is a "Low-Resource" Savior: Self-supervised learning (SSL) provides a robust feature set that prevents the model from "forgetting" what music sounds like when it's only shown a tiny, imbalanced dataset of specific techniques.
  • Pitch and IPT are Inseparable: The "aha moment" here is that IPTs aren't just timbral changes; they are often structural changes to pitch. By forcing the model to learn pitch as an auxiliary task, the IPT detection becomes significantly more accurate.
  • Onsets Anchor Events: Frame-level detection (is this frame a vibrato?) is easier than event-level detection (where does this vibrato start and end?). The onset-triggered post-processing acts as a filter, drastically reducing false positives that plague other models.

Key Results

MERTech outperformed previous state-of-the-art (SOTA) models across the board:

  • Guzheng_Tech99: It improved the event-level micro-F1 score by a staggering 52.8% (from 38.8% to 91.6%) compared to previous multi-scale models.
  • Generalization: Even on the Chinese bamboo flute (CBF), where no pitch labels were available for finetuning, the basic "IPT finetune" version of the model outperformed existing specialized architectures.
  • Ablation Success: The tests showed that the "Full MERTech" (IPT + Pitch + Onset + Post-processing) was consistently superior to versions that dropped the auxiliary tasks.

Limitations and Future Work

While highly effective, the model still relies on a manually set threshold (0.5) for the onset detector, which might not be optimal for every instrument or recording environment.

The authors suggest that future work should explore semi-supervised learning—finding ways to use even more unlabeled data during the finetuning stage—and extending the model to even more "low-resource" musical styles that are currently underrepresented in AI research.

Summary updated

Abstract

Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class. 1
BibTeX
@inproceedings{li2024mertech,
  author = {Dichucheng Li and Yinghao Ma and Weixing Wei and Qiuqiang Kong and Yulun Wu and Mingjin Che and Fan Xia and Emmanouil Benetos and Wei Li},
  title = {{Mertech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model with Multi-Task Finetuning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=PomBYCC2ND},
}

Loading related content...

Figure