Abstract
Instrument recognition is a fundamental task in music information retrieval, yet little has been done to predict the presence of instruments in multi-instrument music for each time frame. This task is important for not only automatic transcription but also many retrieval problems. In this paper, we use the newly released MusicNet dataset to study this front, by building and evaluating a convolutional neural network for making frame-level instrument prediction. We consider it as a multi-label classification problem for each frame and use frame-level annotations as the supervisory signal in training the network. Moreover, we experiment with different ways to incorporate pitch information to our model, with the premise that doing so informs the model the notes that are active per frame, and also encourages the model to learn relative rates of energy buildup in the harmonic partials of different instruments. Experiments show salient performance improvement over baseline methods. We also report an analysis probing how pitch information helps the instrument prediction task.
The Challenge of Polyphonic Recognition
In the field of Music Information Retrieval (MIR), traditional instrument recognition systems often focus on identifying the "predominant" instrument in a clip or work exclusively with solo recordings. However, real-world music is inherently polyphonic. To advance applications like automatic transcription or granular search (e.g., identifying the entry of a specific instrument), systems must achieve frame-level recognition—the ability to identify every active instrument at every millisecond.
This task faces two primary hurdles:
- Data Scarcity: Obtaining frame-level labels for multi-instrument recordings is labor-intensive.
- Timbral Overlap: Instruments within the same family, such as the violin and viola, possess remarkably similar spectral profiles, making them difficult to distinguish using raw frequency data alone.
Pitch-Aware Deep Learning
To address these challenges, we propose a pitch-aware approach using the MusicNet dataset, which provides over 1 million precise frame-level labels for classical recordings. Our model utilizes the Constant-Q Transform (CQT) as its primary input, as it aligns well with musical scales.
The core innovation is the Harmonic Series Feature (HSF). Rather than expecting a standard CNN to implicitly learn the relationship between a fundamental frequency ($f_0$) and its overtones, we explicitly provide this structure. The HSF mechanism creates a "harmonic map" by shifting active pitch entries upwards by multiples of the fundamental frequency ($2f_0, 3f_0$, etc.). These maps are concatenated with the CQT and processed through a CNN with Residual Blocks.
Key Insights: Pitch as a Timbre Multiplier
Our experiments reveal that timbre and pitch are deeply intertwined. An instrument's unique "fingerprint" is defined by how energy is distributed across its harmonics. By explicitly aligning the model's attention to these harmonics via HSF, the network learns the distinct "energy buildup" rates of different instruments.
Interestingly, not all harmonics are equally beneficial. While instruments like the Bassoon and Horn performed best when the model analyzed up to five harmonics (HSF-5), String instruments often fared better with fewer. This suggests that for strings, deeper harmonics may introduce noise that obscures their already similar lower partials.
Results and Generalization
The integration of pitch information yielded significant performance gains:
- Performance Boost: The best pitch-aware model (using ground-truth pitch) achieved an average F1-score of 0.933, compared to the CQT-only baseline of 0.887.
- Robustness: Even with estimated pitches, the HSF-5 model outperformed the baseline with an F1-score of 0.896.
- Targeted Improvements: The most dramatic gains occurred with difficult-to-distinguish instruments. For the Viola, the F1-score rose from 0.830 to 0.916.
Qualitative tests on non-classical tracks (e.g., Adele's Make You Feel My Love) demonstrated the model's ability to generalize, correctly identifying piano and strings while ignoring irrelevant categories like brass or woodwinds.
Conclusion and Future Directions
While the HSF approach significantly improves recognition, it remains dependent on the quality of multi-pitch estimation. Future work will explore joint learning frameworks to simultaneously estimate pitch and recognize instruments, as well as the use of recurrent neural networks (RNNs) to better capture the temporal dynamics of musical performance.
Summary updated