vocadito: A dataset of solo vocals with f0, note, and lyric annotations

Rachel M. Bittner, Katherine Pasalo, Juan José Bosch, Gabriel Meseguer-Brocal, David RubinsteinarXiv 2021 · arXiv

Abstract

Vocadito is a new, open-access dataset of solo singing designed to address the inherent challenges of vocal transcription. By providing multiple human annotations for the same audio, the authors demonstrate that defining "ground truth" for musical notes is highly subjective, with expert musicians agreeing on note events only about 64% of the time. This dataset includes frame-level pitch ($f_0$), discrete note events, and lyrics across multiple languages, serving as a unique benchmark for the Music Information Retrieval (MIR) community.

Introduction: The Challenge of Vocal Transcription

Transcribing the human voice is significantly more complex than transcribing mechanical instruments like the piano. While a piano has discrete keys and fixed pitches, a singer produces a continuous frequency stream ($f_0$) characterized by vibrato, glissando, and unpitched transients (such as the "s" in "song").

The research community faces two primary hurdles:

  1. The Subjectivity Gap: Determining when a vocal flourish constitutes a separate note versus an ornament is subjective. Furthermore, the exact onset of a note—whether at the initial consonant or the start of the vowel—is often ambiguous.
  2. Data Scarcity: Most existing datasets lack the "holy trinity" of annotations: frame-level pitch ($f_0$), discrete note events, and lyrics. Moreover, few provide multiple independent human annotations to account for musical subjectivity.

The vocadito Dataset

The researchers collected 40 short excerpts (10–40 seconds) from 28 volunteers. To ensure real-world applicability, recordings were made on everyday devices like cell phones and laptops rather than in studio environments. The dataset covers seven languages, including English, French, German, and Spanish.

Annotation Process

  • $f_0$ (Pitch): Extracted using the pYIN algorithm and manually corrected by a musician.
  • Notes: Two musicians independently annotated the notes for every track using Tony, a specialized melody transcription tool. They were instructed to annotate what they would play to reproduce the singing on a piano.
  • Lyrics: Transcribed by fluent speakers of the represented languages.
Visualization of $f_0$, notes, and lyrics showing the differences between two annotators.

As shown in the figure above, the discrepancy between annotators often arises from how they handle pitch fluctuations and transitions between words, highlighting the difficulty in establishing a single ground truth.

Key Insights: The Subjectivity of Music

The study reveals that human agreement in vocal transcription is surprisingly low. The F-score between the two expert annotators was only 0.64. This suggests that current evaluation metrics for AI transcription may be penalizing models for "errors" that are actually valid stylistic interpretations.

A common "quantization" approach—rounding $f_0$ to the nearest semitone—performed poorly (F-score of 0.20), proving that note estimation requires an understanding of musical structure and timing beyond simple mathematics. The primary source of disagreement between humans involved "grace notes" and pitch fluctuations; one annotator might split these into multiple short notes, while another groups them into a single, longer note.

Results and Benchmarking

The authors compared vocadito to existing benchmarks like DALI and TONAS. While smaller in scale, vocadito is unique in providing isolated vocals with $f_0$, lyrics, and multi-annotator note labels.

Inter-annotator agreement metrics including Accuracy and F-score.

The table above summarizes the inter-annotator agreement. While Mean Accuracy ($\mu$) is relatively high at 0.83, the F-score is significantly lower at 0.64. This discrepancy occurs because Accuracy is more forgiving of timing offsets than the F-score.

Baseline tests using the crepe algorithm for $f_0$ tracking achieved a Raw Pitch Accuracy of 0.98, suggesting that frequency tracking is largely a solved problem. However, note transcription remains a challenge; the Vocano algorithm achieved a best F-score of only 0.50, significantly lower than human agreement.

Conclusion and Future Work

Vocadito serves as a specialized evaluation set for the MIR community. Future work includes adding syllable-by-syllable time-aligned lyrics to study how phonetic boundaries influence note perception. The authors also advocate for new transcription metrics that can accommodate "multiple correct answers" rather than relying on a single, subjective human reference.

Summary updated

Abstract

To compliment the existing set of datasets, we present a small dataset entitled vocadito, consisting of 40 short excerpts of monophonic singing, sung in 7 different languages by singers with varying of levels of training, and recorded on a variety of devices. We provide several types of annotations, including $f_0$, lyrics, and two different note annotations. All annotations were created by musicians. We provide an analysis of the differences between the two note annotations, and see that the agreement level is low, which has implications for evaluating vocal note estimation algorithms. We also analyze the relation between the $f_0$ and note annotations, and show that quantizing $f_0$ values in frequency does not provide a reasonable note estimate, reinforcing the difficulty of the note estimation task for singing voice. Finally, we provide baseline results from recent algorithms on vocadito for note and $f_0$ transcription. Vocadito is made freely available for public use.
BibTeX
@misc{bittner2021vocadito,
  author = {Rachel M. Bittner and Katherine Pasalo and Juan José Bosch and Gabriel Meseguer-Brocal and David Rubinstein},
  title = {{vocadito: A dataset of solo vocals with f0, note, and lyric annotations}},
  year = {2021},
  eprint = {2110.05580},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.05580},
}
Figure