Towards Automatic Transcription of Polyphonic Electric Guitar Music: A New Dataset and a Multi-Loss Transformer Model

Yu-Hua Chen, Wen-Yi Hsiao, Tsu-Kuang Hsieh, Jyh-Shing Roger Jang, Yi-Hsuan Yang

TL;DR

Researchers have introduced EGDB, a new dataset for electric guitar transcription that accounts for the instrument's diverse tonal range by re-rendering performances through various amplifiers. They also proposed a multi-loss Transformer model that outperforms standard sequence-to-sequence models by combining symbolic output losses with frame-level acoustic supervision.

What Problem Does This Paper Address?

Automatic Music Transcription (AMT)—converting audio into sheet music or MIDI—is significantly more advanced for piano than for other instruments. This is largely due to data: piano labels are easy to generate via MIDI-enabled keyboards, leading to massive datasets like MAESTRO (~200 hours).

Electric guitar transcription is a "boss level" challenge for two reasons:

  1. The Timbre Problem: Unlike a piano, an electric guitar’s sound changes drastically depending on the amplifier and distortion settings. A model trained on "clean" guitar often fails on a "high-gain" heavy metal lead.
  2. Data Scarcity: Existing datasets like GuitarSet are small (30 minutes). Manually annotating polyphonic guitar music is grueling and slow.

This paper aims to bridge the performance gap between piano and guitar AMT by providing more diverse data and a more robust model architecture.

Methods

The authors developed a Multi-loss seq2seq Transformer.

Standard Transformer-based transcription models (like those used for piano) usually measure loss only at the output of the decoder—the "symbolic space" where note events are predicted. However, the authors found that for guitar, the model benefits from "hints" earlier in the process.

Their architecture adds three auxiliary "stacks" to the encoder:

  • Onset Stack: Predicts if a note is starting.
  • Offset Stack: Predicts if a note is ending.
  • Activation Stack: Predicts if a note is currently being held.

By calculating losses on these stacks (the "Encoder Output") in addition to the final "Event Loss" from the decoder, the model receives stronger supervisory signals during training, which is particularly helpful when the dataset is relatively small.

Datasets

  • EGDB (Electric Guitar DataBase): The star of the paper. It contains 118 minutes of unique performances (240 tablatures).
  • The "Re-rendering" Trick: They recorded the "Direct Input" (DI) signal using a hexaphonic pickup (one signal per string). They then used software (Guitar Rig 5) to re-render that clean audio through five different classic amp simulations (e.g., Marshall, Fender, Mesa Boogie). This effectively multiplied the dataset's diversity without requiring extra recording time.
  • GuitarSet: An existing, smaller dataset used for context and comparison.
  • YouTube Samples: Real-world recordings used to test the model's "in the wild" generalization.

Key Insights and Ideas

  • Timbre Matters More Than You Think: Models trained only on clean (DI) signals perform poorly on distorted audio. However, training on a mix of just three different "timbres" (amps) significantly improved the model's ability to transcribe completely unseen guitar tones.
  • The Encoder Advantage: In their experiments, the "Encoder Output" (the frame-wise predictions) actually yielded better transcription results than the "Decoder Output" (the sequence of tokens). This suggests that for guitar, the acoustic frame-level features are currently more reliable than the model's internal "language model" of note sequences.
  • The "Piano Gap": Even with these improvements, guitar transcription (Onset F1 ~0.61) still lags far behind piano transcription (Onset F1 ~0.95). This highlights how much harder it is to untangle the overlapping harmonics of a distorted guitar.

Key Results

  • Performance: The proposed multi-loss Transformer achieved an Onset F1 score of 0.613 on the EGDB-DI test set, outperforming the standard CE-only Transformer (0.568).
  • Generalization: When tested on "unseen" amplifiers, the model trained on multiple timbres (3Amps) maintained an Onset F1 of 0.592, whereas the model trained only on clean DI signals dropped significantly.
  • Real-World Testing: On manually annotated YouTube clips, the model achieved an Onset F1 of 0.550, proving it can handle professional recordings with effects like delay and reverb, even if they weren't in the training set.

Limitations and Future Work

  • Note Offsets: The authors admit that detecting exactly when a guitar note ends is still very difficult, partly because their automated annotation process for offsets is less reliable than for onsets.
  • Tablature vs. Sheet Music: This paper focuses on "sheet transcription" (what notes are played). It does not yet tackle "tab transcription" (which specific string and fret were used), which is vital for guitarists.
  • Playing Techniques: Complex techniques like bends, slides, and vibrato are not yet captured in the transcription output. The authors suggest adding specific "prediction heads" for these techniques in future iterations.

Summary updated

Abstract

In this paper, we propose a new dataset named EGDB, that contains transcriptions of the electric guitar performance of 240 tablatures rendered with different tones. Moreover, we benchmark the performance of two well-known transcription models proposed originally for the piano on this dataset, along with a multi-loss Transformer model that we newly propose. Our evaluation on this dataset and a separate set of real-world recordings demonstrate the influence of timbre on the accuracy of guitar sheet transcription, the potential of using multiple losses for Transformers, as well as the room for further improvement for this task.
BibTeX
@inproceedings{chen2022towards,
  author = {Yu-Hua Chen and Wen-Yi Hsiao and Tsu-Kuang Hsieh and Jyh-Shing Roger Jang and Yi-Hsuan Yang},
  title = {{Towards Automatic Transcription of Polyphonic Electric Guitar Music: A New Dataset and a Multi-Loss Transformer Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=8KpmbEAsic},
}

Loading related content...

Figure