Introduction
In musical compositions, the "lead" instrument is the focal point that commands the listener's attention, such as a vocal melody or a featured solo. While humans intuitively identify these elements, automating Lead Instrument Detection (LID) is a complex task. Traditional methods often rely on analyzing the final audio mixture, which obscures the individual contributions of specific instruments. This research introduces a novel framework for multitrack LID, processing synchronized individual tracks to identify the lead instrument with high precision.
Methodology: Track-wise Attention
The proposed architecture leverages self-supervised learning and a specialized attention mechanism to handle an arbitrary number of input tracks:
- Feature Extraction: Each track, including the full mixture, is processed through MERT, a music-specific self-supervised model. This extracts high-dimensional features capturing timbre and pitch.
- Attention Mechanism: The core innovation is a track-wise attention module. The model treats the global mixture as a query and the individual instrument tracks as keys and values. By calculating the dot product between the mixture and each track, the system determines the relative "auditory importance" of each instrument at any given time.
- Classification Strategy: The model can perform both instrument classification (e.g., "piano") and track classification (e.g., "Track 3"). The latter approach allows the system to generalize to unseen instruments by identifying the lead index rather than a predefined label.
Experimental Setup
The framework was evaluated using two datasets: the MJN (Internal) dataset, featuring 7.10 hours of live recordings with realistic microphone "bleeding," and MedleyDB, a public dataset used to assess cross-domain generalization. To ensure the model relies on audio content rather than metadata, the researchers employed permutation augmentation, randomly shuffling track IDs during training.
Results and Analysis
The experimental results demonstrate that the track-wise attention mechanism is superior to simpler aggregation methods. As shown in the performance metrics, the "Track attn." model significantly outperforms "Track avg." (averaging features) and "From mix" (analyzing only the final mixture).
Key Findings
- Superiority over Baselines: The model outperformed traditional CRNN and SVM baselines. For segment-level guitar solo detection, it achieved an 87.50% F1 score, compared to just 20.80% for the SVM.
- Generalization: The track classification approach proved highly robust. When tested on unseen instruments like the organ, the model successfully identified them as leads, showing a +16.45% improvement in F1 score over standard instrument classification in out-of-domain tests.
- Fine-tuning Necessity: Ablation studies confirmed that full parameter fine-tuning of the MERT encoder is essential; freezing the encoder resulted in a drastic performance drop of over 40% in some metrics.
Conclusion and Future Work
This multitrack LID framework represents a significant step forward in automated music analysis. By mimicking human "selective hearing" through mixture-queried attention, it achieves state-of-the-art results. While currently limited to identifying a single lead instrument and requiring multitrack inputs, future work may extend this to multi-lead scenarios and integration with source separation tools for use on standard audio files. Potential applications include intelligent automatic mixing and automated generation of song previews.
Summary updated