towards automatic transcription of polyphonic electric guitar music: a new dataset and a multi-loss transformer model

Yu-Hua Chen, Wen-Yi Hsiao, Tsu-Kuang Hsieh, Jyh-Shing Roger Jang, Yi-Hsuan Yang

Abstract

In this paper, we propose a new dataset named EGDB, that contains transcriptions of the electric guitar performance of 240 tablatures rendered with different tones. Moreover, we benchmark the performance of two well-known transcription models proposed originally for the piano on this dataset, along with a multi-loss Transformer model that we newly propose. Our evaluation on this dataset and a separate set of real-world recordings demonstrate the influence of timbre on the accuracy of guitar sheet transcription, the potential of using multiple losses for Transformers, as well as the room for further improvement for this task.

Introduction

Automatic Music Transcription (AMT)—the process of converting raw audio into symbolic representations like sheet music or MIDI—has seen significant success in the domain of piano music. This progress is largely attributed to the availability of massive, high-quality datasets like MAESTRO, where labels are easily generated via MIDI-enabled instruments. However, electric guitar transcription remains a significant challenge due to two primary factors: the extreme variability of timbre caused by amplification and effects, and the scarcity of large-scale annotated polyphonic datasets.

The EGDB Dataset

To address the data scarcity, the researchers introduced the Electric Guitar DataBase (EGDB). It consists of 118 minutes of unique performances across 240 tablatures. To simulate the tonal diversity of the instrument, the authors employed a "re-rendering" technique. They recorded the clean Direct Input (DI) signal using hexaphonic pickups and then processed this audio through five distinct amplifier simulations (e.g., Marshall, Fender, Mesa Boogie). This approach effectively multiplied the dataset's diversity, allowing models to be trained on a wide spectrum of guitar tones without additional recording sessions.

Proposed Methodology: Multi-Loss Transformer

Standard sequence-to-sequence (seq2seq) Transformer models typically calculate loss only at the decoder's output, where final note events are predicted. For guitar transcription, the authors found that providing intermediate "hints" to the model significantly improves performance.

Architecture of the proposed multi-loss Transformer model, featuring auxiliary stacks for onset, offset, and activation predictions.

As shown in the architecture above, the proposed model adds three auxiliary stacks to the encoder output:

  • Onset Stack: Predicts the start of a note.
  • Offset Stack: Predicts the end of a note.
  • Activation Stack: Predicts if a note is currently sustained.

By combining these frame-level acoustic losses with the final symbolic "Event Loss" from the decoder, the model receives a stronger supervisory signal, which is particularly beneficial when training on relatively small datasets.

Experimental Results and Insights

The proposed multi-loss Transformer was benchmarked against existing models like the Onset and Frame (OAF) model and a standard Cross-Entropy (CE) only Transformer.

Performance comparison of the proposed multi-loss Transformer against baseline models on the EGDB-DI dataset.

Key Findings:

  1. Superior Performance: The multi-loss Transformer achieved a state-of-the-art Onset F1 score of 0.613 on the EGDB-DI test set, outperforming the standard CE-only Transformer (0.568).
  2. The Timbre Advantage: Training on a mix of just three different amplifier tones (3Amps) significantly improved generalization. When tested on unseen amplifiers, the 3Amps-trained model maintained an Onset F1 of 0.592, whereas models trained only on clean DI signals saw a sharp performance drop.
  3. Real-World Generalization: In tests on manually annotated YouTube clips featuring professional recordings with complex effects like delay and reverb, the model achieved an Onset F1 of 0.550, demonstrating its robustness in "in the wild" scenarios.
  4. Encoder vs. Decoder: Interestingly, the frame-wise predictions from the "Encoder Output" currently yield more reliable results than the "Decoder Output." This suggests that for guitar music, acoustic frame-level features are currently more robust than the model's internal language model of note sequences.

Conclusion and Future Work

While the proposed model and dataset represent a significant step forward, a substantial gap remains between guitar (Onset F1 ~0.61) and piano transcription (Onset F1 ~0.95). Future work will focus on improving note offset detection, which remains difficult due to the nature of guitar decay. Additionally, the researchers aim to expand the model to include guitar-specific techniques such as bends, slides, and vibrato, as well as full tablature transcription (identifying specific strings and frets).

Summary updated

Abstract

In this paper, we propose a new dataset named EGDB, that con-tains transcriptions of the electric guitar performance of 240 tab-latures rendered with different tones. Moreover, we benchmark theperformance of two well-known transcription models proposed orig-inally for the piano on this dataset, along with a multi-loss Trans-former model that we newly propose. Our evaluation on this datasetand a separate set of real-world recordings demonstrate the influenceof timbre on the accuracy of guitar sheet transcription, the potentialof using multiple losses for Transformers, as well as the room forfurther improvement for this task.
BibTeX
@inproceedings{chen2022towards,
  author = {Yu-Hua Chen and Wen-Yi Hsiao and Tsu-Kuang Hsieh and Jyh-Shing Roger Jang and Yi-Hsuan Yang},
  title = {{towards automatic transcription of polyphonic electric guitar music: a new dataset and a multi-loss transformer model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=8KpmbEAsic},
}
Figure