Abstract
Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class.
Introduction
In music information retrieval (MIR), detecting Instrument Playing Techniques (IPTs)—such as vibratos, glissandos, or hammer-ons—is a significant challenge. Unlike basic instrument identification, IPT detection requires capturing subtle nuances in timbre and pitch. Two major hurdles hinder progress: data scarcity, as labeling IPTs requires expert knowledge, and class imbalance, where specific techniques are rare compared to "normal" notes.
Methodology: The MERTech Framework
The proposed MERTech framework addresses these challenges by leveraging transfer learning from a large-scale self-supervised model, MERT-v1-95M. MERT is pre-trained on 160,000 hours of unlabeled music, allowing it to learn a rich representation of musical "language."
Multi-Task Finetuning
Rather than training solely for IPT detection, MERTech employs multi-task finetuning with two auxiliary tasks: pitch detection and IPT onset detection. The rationale is that IPTs are often defined by pitch movement (e.g., glissandos), and knowing the exact start (onset) of a technique helps localize it in time.
Architecture and Post-Processing
As shown in Figure 1, the model takes raw waveforms and processes them through MERT's Transformer encoder layers. A learnable weighted sum of these layers is fed into two branches. To improve event-level accuracy, a post-processing "gate" is used: an IPT activation is only considered a new event if the onset detector also triggers a signal in the same frame.
Experiments and Results
The researchers evaluated MERTech on three diverse benchmarks: Guzheng_Tech99 (Chinese zither), EG-Solo (electric guitar), and CBFdataset (Chinese bamboo flute).
MERTech achieved state-of-the-art (SOTA) performance across all datasets. Notably, on the Guzheng_Tech99 dataset, the event-level micro-F1 score improved from 38.8% (previous SOTA) to 91.6%.
Key Insights
- SSL as a Foundation: Self-supervised pre-training provides a robust feature set that prevents overfitting on small, imbalanced IPT datasets.
- Auxiliary Tasks Matter: Forcing the model to learn pitch and onsets significantly improves IPT detection accuracy, as these features are intrinsically linked to how techniques are performed.
- Onset Gating: The onset-triggered post-processing acts as a powerful filter, drastically reducing false positives in event-level detection.
Conclusion
MERTech demonstrates that self-supervised models pre-trained on massive unlabeled data can be effectively adapted for specialized, low-resource MIR tasks. By integrating pitch and onset detection as auxiliary tasks, MERTech sets a new standard for IPT detection across various instruments and musical traditions. Future work will explore semi-supervised learning to further leverage unlabeled data during the finetuning phase.
Summary updated