Abstract
In this paper, we propose a new dataset named EGDB, that contains transcriptions of the electric guitar performance of 240 tablatures rendered with different tones. Moreover, we benchmark the performance of two well-known transcription models proposed originally for the piano on this dataset, along with a multi-loss Transformer model that we newly propose. Our evaluation on this dataset and a separate set of real-world recordings demonstrate the influence of timbre on the accuracy of guitar sheet transcription, the potential of using multiple losses for Transformers, as well as the room for further improvement for this task.
Introduction
Automatic Music Transcription (AMT)—the process of converting raw audio into symbolic representations like sheet music or MIDI—has seen significant success in the domain of piano music. This progress is largely attributed to the availability of massive, high-quality datasets like MAESTRO, where labels are easily generated via MIDI-enabled instruments. However, electric guitar transcription remains a significant challenge due to two primary factors: the extreme variability of timbre caused by amplification and effects, and the scarcity of large-scale annotated polyphonic datasets.
The EGDB Dataset
To address the data scarcity, the researchers introduced the Electric Guitar DataBase (EGDB). It consists of 118 minutes of unique performances across 240 tablatures. To simulate the tonal diversity of the instrument, the authors employed a "re-rendering" technique. They recorded the clean Direct Input (DI) signal using hexaphonic pickups and then processed this audio through five distinct amplifier simulations (e.g., Marshall, Fender, Mesa Boogie). This approach effectively multiplied the dataset's diversity, allowing models to be trained on a wide spectrum of guitar tones without additional recording sessions.
Proposed Methodology: Multi-Loss Transformer
Standard sequence-to-sequence (seq2seq) Transformer models typically calculate loss only at the decoder's output, where final note events are predicted. For guitar transcription, the authors found that providing intermediate "hints" to the model significantly improves performance.
As shown in the architecture above, the proposed model adds three auxiliary stacks to the encoder output:
- Onset Stack: Predicts the start of a note.
- Offset Stack: Predicts the end of a note.
- Activation Stack: Predicts if a note is currently sustained.
By combining these frame-level acoustic losses with the final symbolic "Event Loss" from the decoder, the model receives a stronger supervisory signal, which is particularly beneficial when training on relatively small datasets.
Experimental Results and Insights
The proposed multi-loss Transformer was benchmarked against existing models like the Onset and Frame (OAF) model and a standard Cross-Entropy (CE) only Transformer.
Key Findings:
- Superior Performance: The multi-loss Transformer achieved a state-of-the-art Onset F1 score of 0.613 on the EGDB-DI test set, outperforming the standard CE-only Transformer (0.568).
- The Timbre Advantage: Training on a mix of just three different amplifier tones (3Amps) significantly improved generalization. When tested on unseen amplifiers, the 3Amps-trained model maintained an Onset F1 of 0.592, whereas models trained only on clean DI signals saw a sharp performance drop.
- Real-World Generalization: In tests on manually annotated YouTube clips featuring professional recordings with complex effects like delay and reverb, the model achieved an Onset F1 of 0.550, demonstrating its robustness in "in the wild" scenarios.
- Encoder vs. Decoder: Interestingly, the frame-wise predictions from the "Encoder Output" currently yield more reliable results than the "Decoder Output." This suggests that for guitar music, acoustic frame-level features are currently more robust than the model's internal language model of note sequences.
Conclusion and Future Work
While the proposed model and dataset represent a significant step forward, a substantial gap remains between guitar (Onset F1 ~0.61) and piano transcription (Onset F1 ~0.95). Future work will focus on improving note offset detection, which remains difficult due to the nature of guitar decay. Additionally, the researchers aim to expand the model to include guitar-specific techniques such as bends, slides, and vibrato, as well as full tablature transcription (identifying specific strings and frets).
Summary updated