Extended playing techniques: the next milestone in musical instrument recognition

Vincent Lostanlen, Joakim Andén, Mathieu Lagrange

Abstract

The expressive variability in producing a musical note conveys information essential to the modeling of orchestration and style. As such, it plays a crucial role in computer-assisted browsing of massive digital music corpora. Yet, although the automatic recognition of a musical instrument from the recording of a single "ordinary" note is considered a solved problem, automatic identification of instrumental playing technique (IPT) remains largely underdeveloped. We benchmark machine listening systems for query-by-example browsing among 143 extended IPTs for 16 instruments, amounting to 469 triplets of instrument, mute, and technique. We identify and discuss three necessary conditions for significantly outperforming the traditional mel-frequency cepstral coefficient (MFCC) baseline: the addition of second-order scattering coefficients to account for amplitude modulation, the incorporation of long-range temporal dependencies, and metric learning using large-margin nearest neighbors (LMNN) to reduce intra-class variability. Evaluating on the Studio On Line (SOL) dataset, we obtain a precision at rank 5 of 99.7% for instrument recognition (baseline at 89.0%) and of 61.0% for IPT recognition (baseline at 44.5%). We interpret this gain through a qualitative assessment of practical usability and visualization using nonlinear dimensionality reduction.

Introduction

In the field of Music Information Retrieval (MIR), researchers have achieved high accuracy in "organological" classification—identifying the instrument itself (the "what"). However, the "chironomical" aspect—the physical gesture or instrumental playing technique (IPT) representing the "how"—is often overlooked. Standard models frequently treat variations like vibrato, pizzicato, or flatterzunge as noise or intra-class variability to be ignored. This paper addresses this gap by introducing a robust framework for identifying 143 extended IPTs across 16 instruments, moving the benchmark from simple identification to a nuanced understanding of musical gesture.

Methodology

The authors propose a query-by-example system using a $k$-nearest neighbors ($k$-NN) approach, powered by three key technical components:

  1. Wavelet Scattering Transform: Traditional features like Mel-Frequency Cepstral Coefficients (MFCCs) only capture coarse spectral envelopes. By using a second-order scattering transform, the model captures amplitude modulation—the temporal fluctuations that define complex techniques.
  2. Large-Margin Nearest Neighbors (LMNN): This supervised metric learning algorithm learns a transformation matrix that optimizes the feature space, pulling sounds with the same label closer together while pushing dissimilar sounds apart.
  3. Long-range Temporal Dependencies: Increasing the analysis time scale from 25ms to 1 second was found to be crucial for capturing the evolution of complex gestures.

These methods were evaluated using the Studio On Line (SOL) dataset, which includes 25,444 isolated notes and 469 unique combinations of instrument, mute, and technique.

Key Results and Insights

The proposed system significantly outperformed the MFCC baseline across both tasks:

  • Instrument Recognition: Achieved 99.7% precision (P@5), compared to 89.0% for the baseline.
  • IPT Recognition: Reached 61.0% precision, a substantial improvement over the 44.5% baseline.
Visualizing different playing techniques: (i) Articulation (trill) and (j) Phrasing (détaché).

A key insight is the "super-additive" effect: for IPT recognition, improvements from better features and metric learning are not just additive; increasing the time scale only helps significantly when combined with metric learning. Qualitatively, scattering transforms resolve temporal fluctuations that MFCCs cannot, leading to much clearer clustering of different playing techniques as seen in the visual representations of trills and détaché phrasing.

Conclusion and Future Work

This research demonstrates that while instrument recognition is largely solved, IPT recognition requires specialized features and metric learning to handle expressive variability. Future work aims to transition from isolated notes to polyphonic, continuous music, moving toward "captioning" systems that can simultaneously estimate pitch, dynamics, instrument, and technique in complex audio streams.

Summary updated

BibTeX
@inproceedings{lostanlen2018extended,
  author = {Vincent Lostanlen and Joakim Andén and Mathieu Lagrange},
  title = {{Extended playing techniques: the next milestone in musical instrument recognition}},
  booktitle = {Dlfm},
  year = {2018},
  doi = {10.1145/3273024.3273036},
}
Figure