Abstract
As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one's desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference music mixture as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer's activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture of music and retrieve the desired instruments successfully.
The Challenge of Musical Instrument Retrieval
In modern digital music production, finding the "perfect sound" often involves navigating libraries containing thousands of virtual instruments and samples. Currently, producers rely on vague text descriptions (e.g., "warm synth pad") or manual auditioning—a tedious process that hampers creativity.
While Musical Instrument Recognition (identifying broad categories like "guitar" or "piano") is well-established, it lacks the granularity required for production. A producer doesn't just need any guitar; they need the specific timbre heard in a reference track. This paper introduces Musical Instrument Retrieval, moving beyond classification to fine-grained search where an audio mixture serves as the query to find exact matching instruments in a digital library.
Proposed Methodology
The researchers developed a two-stage deep learning architecture based on Convolutional Neural Networks (CNNs) to bridge the gap between complex audio mixtures and individual instrument samples.
1. Single-Instrument Encoder (SIE)
The SIE is trained on single-track audio to classify instruments. However, its primary purpose is to generate a timbral fingerprint. By extracting the 1024-dimensional vector from the network's penultimate layer, the model creates an embedding that represents the unique sonic characteristics of an instrument, allowing it to generalize to instruments it has never seen before.
2. Multi-Instrument Encoder (MIE)
The MIE is designed to process audio mixtures containing multiple instruments. Its goal is to output a set of embeddings corresponding to each individual instrument present in the query.
Permutation Invariant Training (PIT): Training the MIE is complex because mixtures lack a natural instrument order. The authors utilized PIT combined with the Hungarian Algorithm to find the optimal match between predicted embeddings and ground-truth SIE embeddings, minimizing cosine distance regardless of the sequence.
Nlakh: A New Large-Scale Dataset
Existing datasets were found insufficient for this task, leading to the creation of Nlakh. This dataset combines NSynth (individual notes) and Lakh MIDI (multi-track structures) to create a robust training ground:
- Nlakh-single: 1,397 hours of single-track audio across 1,006 instruments.
- Nlakh-multi: 153 hours of mixtures with 2–9 simultaneous instruments. To the authors' knowledge, this represents the largest multi-track music dataset in terms of unique instrument variety and total duration.
Key Insights and Results
- Generalization via Embeddings: By focusing on vector space embeddings rather than fixed labels, the system successfully clusters similar-sounding instruments together, even for "unseen" samples in the validation set.
- On-the-Fly Mixing: Training on static mixtures led to overfitting. The researchers found that randomly mixing 2–9 instruments during training significantly improved the model's robustness and retrieval accuracy.
- Performance Metrics: The SIE achieved an Equal Error Rate (EER) of 0.026, outperforming previous benchmarks. The best-performing MIE model (Large-Random) achieved an instrument-level mean Average Precision (mAP) of 0.760 and a weighted F1 score of 0.712.
Limitations and Future Work
While the system marks a significant step forward, challenges remain. Real-world audio often includes heavy processing—such as distortion, reverb, and delay—which can mask an instrument's raw timbral fingerprint. Future work will focus on expanding instrument diversity beyond the current 1,006 classes and optimizing the computational cost of PIT for mixtures with high instrument counts.
Summary updated