Show Me the Instruments: Musical Instrument Retrieval From Mixture Audio

Kyungsu Kim, Minju Park, Haesun Joung, Yunkee Chae, Yeongbeom Hong, Seonghyeon Go, Kyogu Lee

TL;DR

Researchers have developed a new system for Musical Instrument Retrieval, allowing music producers to find specific virtual instruments or samples by using a snippet of a song as a query. By combining a single-instrument embedding encoder with a multi-instrument extractor trained via permutation-invariant loss, the system can identify and retrieve the closest matching instruments from a massive library, even when they are layered together in a complex audio mixture.

What Problem Does This Paper Address?

In modern digital music production, the "perfect sound" is often hidden within a library of thousands of virtual instruments and samples. Currently, producers find these sounds by either reading vague text descriptions (e.g., "warm synth pad") or manually auditioning samples one by one—a tedious and inefficient process.

While "Musical Instrument Recognition" (identifying that a track contains a "guitar") is a well-studied field, it is too coarse for production. A producer doesn't just want any guitar; they want the specific guitar sound they heard in a reference track. This paper moves from broad classification to fine-grained retrieval, treating audio mixtures as search queries to find the exact constituent instruments in a digital library.

Methods

The authors propose a two-stage deep learning architecture based on Convolutional Neural Networks (CNNs):

  1. Single-Instrument Encoder (SIE): This model is trained on single-track audio to classify instruments. However, the goal isn't the classification label itself, but the embedding—a 1024-dimensional vector from the network's penultimate layer that captures the unique "timbral fingerprint" of an instrument.
  2. Multi-Instrument Encoder (MIE): This model takes a mixture of audio (multiple instruments playing at once) and attempts to output a set of embeddings corresponding to the individual instruments present.

The Secret Sauce: Permutation Invariant Training (PIT) Training the MIE is tricky because a mixture doesn't have a natural "order" of instruments. To solve this, the authors used PIT combined with the Hungarian Algorithm. This allows the model to compare its predicted set of embeddings against the ground-truth set by finding the best possible match between them, regardless of the order, and minimizing the cosine distance.

During Inference, the MIE extracts embeddings from a query mixture. These are then compared against a pre-computed library of embeddings (generated by the SIE) using cosine similarity to retrieve the most similar instruments.

Datasets

The researchers found existing datasets lacking in scale or variety, so they introduced Nlakh:

  • Source: A combination of the NSynth dataset (individual instrument notes) and the Lakh MIDI dataset (multi-track musical structures).
  • Nlakh-single: 1,397 hours of single-track audio across 1,006 instruments.
  • Nlakh-multi: 153 hours of mixtures containing at least two instruments.
  • Significance: To the authors' knowledge, this is the largest multi-track music dataset in terms of the number of unique instruments and total audio duration.

Key Insights and Ideas

  • Embeddings over Labels: By focusing on embeddings rather than fixed categories, the system can generalize to "unseen" instruments. The SIE learns a mapping of timbre that allows it to cluster similar-sounding instruments together in vector space, even if it wasn't trained on that specific sample.
  • On-the-Fly Mixing: The authors discovered that training on a fixed set of mixtures led to overfitting. By "randomly mixing" audio samples on-the-fly during training (selecting 2–9 instruments at random), the model became much more robust and achieved better retrieval performance.
  • Architecture Matters: Moving from a standard CNN to a larger, more modern architecture (inspired by ConvNeXt) significantly improved the model's ability to disentangle complex mixtures.

Key Results

The performance was evaluated using F1 scores and mean Average Precision (mAP):

  • Single-Instrument Accuracy: The SIE achieved an Equal Error Rate (EER) of 0.026, outperforming previous benchmarks on the NSynth dataset (0.031).
  • Retrieval Performance: The "Large-Random" model (large architecture trained with random on-the-fly mixing) was the top performer. It achieved an instrument-level mAP of 0.760 and a weighted F1 score of 0.712.
  • Visual Proof: t-SNE visualizations (dimensionality reduction) showed that the SIE successfully clusters different audio samples of the same instrument together, even for instruments in the validation set that the model never saw during training.

Limitations and Future Work

While the results are promising, the authors note a few areas for improvement:

  • Audio Effects: The current dataset consists of relatively "clean" renders. In real-world scenarios, instruments are often processed with heavy distortion, reverb, or delay, which can drastically alter their timbral fingerprint.
  • Instrument Diversity: While 1,006 instruments is a large start, the professional world uses tens of thousands. Expanding the dataset to include more diverse classes and articulations is a logical next step.
  • Computational Cost: Finding the optimal permutation for PIT can become computationally expensive as the number of simultaneous instruments in a mixture increases.

Summary updated

Abstract

As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one’s desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference mixture audio as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer’s activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture audio and retrieve the desired instruments successfully. The code used for the experiment and audio samples are available at: https://github.com/minju0821/musical_instrument_retrieval
BibTeX
@inproceedings{kim2023show,
  author = {Kyungsu Kim and Minju Park and Haesun Joung and Yunkee Chae and Yeongbeom Hong and Seonghyeon Go and Kyogu Lee},
  title = {{Show Me the Instruments: Musical Instrument Retrieval From Mixture Audio}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=nyz7bf1b8k},
}

Loading related content...

Figure