TL;DR
Researchers from Tencent PCG have released HumTrans, the largest open-source dataset for humming melody transcription to date. Spanning over 56 hours of audio with precise MIDI ground truth, it provides a much-needed benchmark for a task where current state-of-the-art models still struggle significantly.
What Problem Does This Paper Address?
Automatic Music Transcription (AMT) — the process of turning audio into musical notation — has seen massive leaps in performance for instruments like the piano. However, vocal transcription, specifically humming, remains a "boss level" challenge.
Humming is difficult for AI because human voices lack the pitch stability and sharp "onsets" (the start of a note) found in instruments. Furthermore, high-quality data is incredibly scarce. Previous datasets like those from the MIREX challenge are often under 10 hours long or have broken download links. Without a large, diverse, and well-labeled dataset, developing robust humming-to-sheet-music applications has been nearly impossible.
Methods
The authors built a custom web-based pipeline to collect high-quality data systematically:
- Melody Selection: They curated 500 musical compositions across various genres, ensuring the melodies were "hummable" (avoiding extreme pitch jumps or ranges).
- Expert Contributors: Instead of general crowdsourcing, they recruited 10 music majors or proficient instrumentalists to ensure pitch accuracy and rhythmic timing.
- Standardized Recording: Participants hummed using the syllable "Da-Da-Da" while listening to a reference track and viewing a musical score. This specific syllable helps models identify note onsets more clearly than a continuous "Mmm."
- Data Cleaning: Recordings were sampled at 44.1 kHz. The team used DNSMOS (a deep noise suppression mean opinion score) to filter out low-quality audio, keeping only files with a score above 2.8.
Datasets
The primary contribution is the HumTrans dataset:
- Scale: 56.22 hours of audio (14,614 recording files).
- Content: 1,000 music segments derived from 500 compositions.
- Labels: Each recording is paired with a reference MIDI file.
- Diversity: Covers a pitch range of five octaves (C2 to C6) and a BPM range of 52 to 156.
For comparison, the paper references existing datasets like MTG-QBH (0.88 hrs), MIR-QBSH (9.85 hrs), and MLEnd Hums and Whistles (29.77 hrs), noting that HumTrans is nearly double the size of the previous largest collection.
Key Insights and Ideas
- The "Da-Da-Da" Strategy: By instructing participants to use specific syllables, the researchers created a "self-labeled" dataset. The rhythmic consistency of the "D" consonant provides a natural anchor for note onsets without requiring manual frame-by-frame annotation.
- Octave-Invariant Evaluation: The authors argue that for humming, being "off" by an octave shouldn't be penalized as heavily as being off by a semitone. Since different people have different natural vocal ranges, a model that transcribes the correct melody but in a different octave is still practically useful.
- The Difficulty Gap: Even with this high-quality data, current SOTA models perform poorly. This suggests that humming transcription isn't just a "data problem"—it's an algorithmic challenge that requires new architectures to handle the nuances of the human voice.
Key Results
The researchers tested four state-of-the-art (SOTA) vocal transcription models on HumTrans: VOCANO, Sheet Sage, MIR-ST500, and JDC-STP.
- Top Performer: JDC-STP achieved the best results, but the numbers were surprisingly low: a Validation F1-score of 6.74% and a Test F1-score of 5.66%.
- Comparison: Even models pre-trained on singing data (like MIR-ST500) struggled with the humming format.
- Conclusion from Results: The low scores across the board highlight that humming transcription is far from a solved problem, and HumTrans provides the necessary "training ground" for the community to improve these figures.
Limitations and Future Work
- Participant Diversity: While the data is high-quality, it only comes from 10 individuals. Future versions could benefit from a wider variety of vocal timbres and skill levels (including non-musicians).
- Syllable Constraints: The dataset relies on "Da-Da-Da." Real-world humming often involves "Mmm," "Ah," or whistling. The authors note that expanding to these other vocalizations is a logical next step.
- Model Performance: The massive gap between current model performance and a "usable" transcription tool indicates a need for new research into joint pitch and onset detection specifically tuned for the human voice.
Check out the dataset: HumTrans is available on Hugging Face, with baseline code available on GitHub.
Summary updated