Abstract
Vocadito is a new, open-access dataset of solo singing designed to address the inherent challenges of vocal transcription. By providing multiple human annotations for the same audio, the authors demonstrate that defining "ground truth" for musical notes is highly subjective, with expert musicians agreeing on note events only about 64% of the time. This dataset includes frame-level pitch ($f_0$), discrete note events, and lyrics across multiple languages, serving as a unique benchmark for the Music Information Retrieval (MIR) community.
Introduction: The Challenge of Vocal Transcription
Transcribing the human voice is significantly more complex than transcribing mechanical instruments like the piano. While a piano has discrete keys and fixed pitches, a singer produces a continuous frequency stream ($f_0$) characterized by vibrato, glissando, and unpitched transients (such as the "s" in "song").
The research community faces two primary hurdles:
- The Subjectivity Gap: Determining when a vocal flourish constitutes a separate note versus an ornament is subjective. Furthermore, the exact onset of a note—whether at the initial consonant or the start of the vowel—is often ambiguous.
- Data Scarcity: Most existing datasets lack the "holy trinity" of annotations: frame-level pitch ($f_0$), discrete note events, and lyrics. Moreover, few provide multiple independent human annotations to account for musical subjectivity.
The vocadito Dataset
The researchers collected 40 short excerpts (10–40 seconds) from 28 volunteers. To ensure real-world applicability, recordings were made on everyday devices like cell phones and laptops rather than in studio environments. The dataset covers seven languages, including English, French, German, and Spanish.
Annotation Process
- $f_0$ (Pitch): Extracted using the pYIN algorithm and manually corrected by a musician.
- Notes: Two musicians independently annotated the notes for every track using Tony, a specialized melody transcription tool. They were instructed to annotate what they would play to reproduce the singing on a piano.
- Lyrics: Transcribed by fluent speakers of the represented languages.
As shown in the figure above, the discrepancy between annotators often arises from how they handle pitch fluctuations and transitions between words, highlighting the difficulty in establishing a single ground truth.
Key Insights: The Subjectivity of Music
The study reveals that human agreement in vocal transcription is surprisingly low. The F-score between the two expert annotators was only 0.64. This suggests that current evaluation metrics for AI transcription may be penalizing models for "errors" that are actually valid stylistic interpretations.
A common "quantization" approach—rounding $f_0$ to the nearest semitone—performed poorly (F-score of 0.20), proving that note estimation requires an understanding of musical structure and timing beyond simple mathematics. The primary source of disagreement between humans involved "grace notes" and pitch fluctuations; one annotator might split these into multiple short notes, while another groups them into a single, longer note.
Results and Benchmarking
The authors compared vocadito to existing benchmarks like DALI and TONAS. While smaller in scale, vocadito is unique in providing isolated vocals with $f_0$, lyrics, and multi-annotator note labels.
The table above summarizes the inter-annotator agreement. While Mean Accuracy ($\mu$) is relatively high at 0.83, the F-score is significantly lower at 0.64. This discrepancy occurs because Accuracy is more forgiving of timing offsets than the F-score.
Baseline tests using the crepe algorithm for $f_0$ tracking achieved a Raw Pitch Accuracy of 0.98, suggesting that frequency tracking is largely a solved problem. However, note transcription remains a challenge; the Vocano algorithm achieved a best F-score of only 0.50, significantly lower than human agreement.
Conclusion and Future Work
Vocadito serves as a specialized evaluation set for the MIR community. Future work includes adding syllable-by-syllable time-aligned lyrics to study how phonetic boundaries influence note perception. The authors also advocate for new transcription metrics that can accommodate "multiple correct answers" rather than relying on a single, subjective human reference.
Summary updated