MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuning

Dichucheng Li, Yinghao Ma, Weixing Wei, Qiuqiang Kong, Yulun Wu, Mingjin Che, Fan Xia, Emmanouil Benetos, Wei Li

Abstract

Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class.

Introduction

In music information retrieval (MIR), detecting Instrument Playing Techniques (IPTs)—such as vibratos, glissandos, or hammer-ons—is a significant challenge. Unlike basic instrument identification, IPT detection requires capturing subtle nuances in timbre and pitch. Two major hurdles hinder progress: data scarcity, as labeling IPTs requires expert knowledge, and class imbalance, where specific techniques are rare compared to "normal" notes.

Methodology: The MERTech Framework

The proposed MERTech framework addresses these challenges by leveraging transfer learning from a large-scale self-supervised model, MERT-v1-95M. MERT is pre-trained on 160,000 hours of unlabeled music, allowing it to learn a rich representation of musical "language."

Multi-Task Finetuning

Rather than training solely for IPT detection, MERTech employs multi-task finetuning with two auxiliary tasks: pitch detection and IPT onset detection. The rationale is that IPTs are often defined by pitch movement (e.g., glissandos), and knowing the exact start (onset) of a technique helps localize it in time.

Figure 1: The MERTech architecture. It uses a MERT backbone with a learnable weighted sum of layers feeding into specialized MLP branches for IPT, pitch, and onset detection.

Architecture and Post-Processing

As shown in Figure 1, the model takes raw waveforms and processes them through MERT's Transformer encoder layers. A learnable weighted sum of these layers is fed into two branches. To improve event-level accuracy, a post-processing "gate" is used: an IPT activation is only considered a new event if the onset detector also triggers a signal in the same frame.

Experiments and Results

The researchers evaluated MERTech on three diverse benchmarks: Guzheng_Tech99 (Chinese zither), EG-Solo (electric guitar), and CBFdataset (Chinese bamboo flute).

MERTech achieved state-of-the-art (SOTA) performance across all datasets. Notably, on the Guzheng_Tech99 dataset, the event-level micro-F1 score improved from 38.8% (previous SOTA) to 91.6%.

Table 1: Performance comparison of MERTech against previous SOTA models. MERTech shows significant gains in both frame-level and event-level metrics.

Key Insights

  • SSL as a Foundation: Self-supervised pre-training provides a robust feature set that prevents overfitting on small, imbalanced IPT datasets.
  • Auxiliary Tasks Matter: Forcing the model to learn pitch and onsets significantly improves IPT detection accuracy, as these features are intrinsically linked to how techniques are performed.
  • Onset Gating: The onset-triggered post-processing acts as a powerful filter, drastically reducing false positives in event-level detection.

Conclusion

MERTech demonstrates that self-supervised models pre-trained on massive unlabeled data can be effectively adapted for specialized, low-resource MIR tasks. By integrating pitch and onset detection as auxiliary tasks, MERTech sets a new standard for IPT detection across various instruments and musical traditions. Future work will explore semi-supervised learning to further leverage unlabeled data during the finetuning phase.

Summary updated

Abstract

Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class.
BibTeX
@inproceedings{li2024mertech,
  author = {Dichucheng Li and Yinghao Ma and Weixing Wei and Qiuqiang Kong and Yulun Wu and Mingjin Che and Fan Xia and Emmanouil Benetos and Wei Li},
  title = {{MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=PomBYCC2ND},
}
Figure