Abstract Traditional metrics for Automatic Music Transcription (AMT), such as the F-measure, fail to account for the varying psychological impact of different transcription errors. This paper presents a methodology for weighting transcription errors based on human perception. Through subjective listening tests, we quantify the "discomfort" caused by specific mistakes—such as pitch inaccuracies versus timing shifts—to develop a Perceptive F-measure that more accurately reflects human musical judgment.
Introduction
Automatic Music Transcription (AMT) aims to convert raw audio into symbolic notation, typically MIDI. Historically, these systems have been evaluated using the F-measure, a mathematical metric that counts correct notes (True Positives), extra notes (False Positives), and missing notes (False Negatives).
However, the F-measure treats every error as equal. In a musical context, a piano note ending 50ms early is often imperceptible, whereas a "clinker" (a wrong pitch) can be jarring. This research bridges the gap between signal processing metrics and human perception by quantifying "discomfort" to create more meaningful evaluation tools.
Methodology
The researchers conducted two primary subjective listening tests with 37 participants (24 musicians and 13 non-musicians):
- Error Ranking: Participants performed pairwise comparisons of audio clips. One was an original piano recording, and the others were "artificial" transcriptions containing a single error type (e.g., an octave error or a timing shift). Using the Bradley-Terry-Luce (BTL) model, these comparisons were converted into a unified scale of perceptual discomfort.
- System Evaluation: Participants rated transcriptions from four real-world AMT systems.
- The Perceptive Metric: Discomfort scores from the first task were used to derive weighting coefficients ($\alpha$) for different error types. These were integrated into a modified Perceptive F-measure and a Perceptive Proportional Transportation Distance (PTD).
The study utilized classical piano excerpts from Chopin, Debussy, Mozart, and J.S. Bach, providing a controlled environment for high-quality perceptual testing.
Key Insights: The Hierarchy of Discomfort
The listening tests revealed a clear hierarchy in how humans perceive transcription mistakes.
- Pitch is King: Pitch errors cause significantly more discomfort than timing or loudness errors. However, harmonicity matters; an octave error is perceived as much less annoying than a "fifth" error or a random pitch replacement.
- Onsets Over Durations: Humans are highly sensitive to when a note starts (onset), but remarkably forgiving about when it ends (offset) on a piano. This is likely due to the piano's natural decay, which makes the exact end of a note less defined.
- The "Ghost Note" Penalty: Inserting extra notes (False Positives) is generally more distracting than missing notes (False Negatives).
- Universal Perception: Interestingly, musicians and non-musicians showed nearly identical trends in their discomfort ratings, suggesting these priorities are universal.
Results and Validation
The authors derived specific weights for the new metric: onset errors were weighted at 0.46, while duration errors were weighted at only 0.03. Random pitch errors (0.29) were nearly twice as annoying as octave errors (0.17).
When applied to real-world system outputs, the Perceptive F-measure scores aligned much more closely with subjective human ratings than the standard F-measure. This validates that a weighted metric can effectively predict how a human listener will evaluate an AI's transcription performance.
Conclusion and Future Work
While this study successfully demonstrates the value of perceptually-weighted metrics, it is currently limited to piano music. Because the piano is a percussive instrument, duration errors are less noticeable than they might be for "forced-vibration" instruments like the violin. Future research will explore non-linear relationships in discomfort—such as the "tipping point" where a few errors become a total failure—and how combinations of different error types interact perceptually.
Summary updated