Humtrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond

Shansong Liu, Xu Li, Dian Li, Ying Shan

TL;DR

Researchers from Tencent PCG have released HumTrans, the largest open-source dataset for humming melody transcription to date. Spanning over 56 hours of audio with precise MIDI ground truth, it provides a much-needed benchmark for a task where current state-of-the-art models still struggle significantly.

What Problem Does This Paper Address?

Automatic Music Transcription (AMT) — the process of turning audio into musical notation — has seen massive leaps in performance for instruments like the piano. However, vocal transcription, specifically humming, remains a "boss level" challenge.

Humming is difficult for AI because human voices lack the pitch stability and sharp "onsets" (the start of a note) found in instruments. Furthermore, high-quality data is incredibly scarce. Previous datasets like those from the MIREX challenge are often under 10 hours long or have broken download links. Without a large, diverse, and well-labeled dataset, developing robust humming-to-sheet-music applications has been nearly impossible.

Methods

The authors built a custom web-based pipeline to collect high-quality data systematically:

  1. Melody Selection: They curated 500 musical compositions across various genres, ensuring the melodies were "hummable" (avoiding extreme pitch jumps or ranges).
  2. Expert Contributors: Instead of general crowdsourcing, they recruited 10 music majors or proficient instrumentalists to ensure pitch accuracy and rhythmic timing.
  3. Standardized Recording: Participants hummed using the syllable "Da-Da-Da" while listening to a reference track and viewing a musical score. This specific syllable helps models identify note onsets more clearly than a continuous "Mmm."
  4. Data Cleaning: Recordings were sampled at 44.1 kHz. The team used DNSMOS (a deep noise suppression mean opinion score) to filter out low-quality audio, keeping only files with a score above 2.8.

Datasets

The primary contribution is the HumTrans dataset:

  • Scale: 56.22 hours of audio (14,614 recording files).
  • Content: 1,000 music segments derived from 500 compositions.
  • Labels: Each recording is paired with a reference MIDI file.
  • Diversity: Covers a pitch range of five octaves (C2 to C6) and a BPM range of 52 to 156.

For comparison, the paper references existing datasets like MTG-QBH (0.88 hrs), MIR-QBSH (9.85 hrs), and MLEnd Hums and Whistles (29.77 hrs), noting that HumTrans is nearly double the size of the previous largest collection.

Key Insights and Ideas

  • The "Da-Da-Da" Strategy: By instructing participants to use specific syllables, the researchers created a "self-labeled" dataset. The rhythmic consistency of the "D" consonant provides a natural anchor for note onsets without requiring manual frame-by-frame annotation.
  • Octave-Invariant Evaluation: The authors argue that for humming, being "off" by an octave shouldn't be penalized as heavily as being off by a semitone. Since different people have different natural vocal ranges, a model that transcribes the correct melody but in a different octave is still practically useful.
  • The Difficulty Gap: Even with this high-quality data, current SOTA models perform poorly. This suggests that humming transcription isn't just a "data problem"—it's an algorithmic challenge that requires new architectures to handle the nuances of the human voice.

Key Results

The researchers tested four state-of-the-art (SOTA) vocal transcription models on HumTrans: VOCANO, Sheet Sage, MIR-ST500, and JDC-STP.

  • Top Performer: JDC-STP achieved the best results, but the numbers were surprisingly low: a Validation F1-score of 6.74% and a Test F1-score of 5.66%.
  • Comparison: Even models pre-trained on singing data (like MIR-ST500) struggled with the humming format.
  • Conclusion from Results: The low scores across the board highlight that humming transcription is far from a solved problem, and HumTrans provides the necessary "training ground" for the community to improve these figures.

Limitations and Future Work

  • Participant Diversity: While the data is high-quality, it only comes from 10 individuals. Future versions could benefit from a wider variety of vocal timbres and skill levels (including non-musicians).
  • Syllable Constraints: The dataset relies on "Da-Da-Da." Real-world humming often involves "Mmm," "Ah," or whistling. The authors note that expanding to these other vocalizations is a logical next step.
  • Model Performance: The massive gap between current model performance and a "usable" transcription tool indicates a need for new research into joint pitch and onset detection specifically tuned for the human voice.

Check out the dataset: HumTrans is available on Hugging Face, with baseline code available on GitHub.

Summary updated

Abstract

This paper introduces the HumTrans dataset, which is publicly available and primarily designed for humming melody transcription. The dataset can also serve as a foundation for downstream tasks such as humming melody based music generation. It consists of 500 musical compositions of different genres and languages, with each composition divided into multiple segments. In total, the dataset comprises 1000 music segments. To collect this humming dataset, we employed 10 college students, all of whom are either music majors or proficient in playing at least one musical instrument. Each of them hummed every segment twice using the web recording interface provided by our designed website 1 . The humming recordings were sampled at a frequency of 44,100 Hz. During the humming session, the main interface provides a musical score for students to reference, with the melody audio playing simultaneously to aid in capturing both melody and rhythm. The dataset encompasses approximately 56.22 hours of audio, making it the largest known humming dataset to date. The dataset will be released on Hugging Face 2 , and we will provide a GitHub repository containing baseline results and evaluation codes 3 .
BibTeX
@inproceedings{liu2024humtrans,
  author = {Shansong Liu and Xu Li and Dian Li and Ying Shan},
  title = {{Humtrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10445818},
}

Loading related content...

Figure