Introduction
The sustain pedal is often described as the "soul" of the piano, providing essential control over resonance and tonal color. While modern AI-driven piano transcription systems have advanced significantly, most still treat the sustain pedal as a simple binary on/off switch. Recent high-resolution models attempt to estimate continuous pedal depth, yet they are typically evaluated using generic signal-processing metrics like Mean Squared Error (MSE) or F1-scores.
Researchers from McGill, NYU, and MBZUAI argue that these standard metrics are "musically deaf." A minor timing shift or signal jitter might lead to a poor MSE score even if the musical intent remains intact. Conversely, a model might achieve a low MSE while failing to capture critical structural gestures, such as a quick "catch" pedal. To address this, the authors propose a new evaluation framework that aligns with piano pedagogy and performance practice.
The Three-Level Evaluation Framework
The core contribution of this work is a multi-tiered approach to evaluating pedal estimation models, moving from granular signal data to high-level musical structures.
- Frame-level (Standard): This level utilizes traditional per-frame error metrics, including MSE, Mean Absolute Error (MAE), and classification accuracy for both binary and 4-class states.
- Action-level: The pedal signal is segmented into three functional states—press, hold, and release. This is achieved by applying linear regression on sliding windows to identify slopes, evaluating whether a model correctly identifies the intent of the pianist's foot movement.
- Gesture-level: Complete press-release cycles are grouped into four canonical "shapes" inspired by piano pedagogy:
- Pinnacle: Short duration, high depth (quick accents).
- Hill: Short duration, low depth (subtle decorative shaping).
- Highland: Long duration, high depth (building resonance).
- Mountain: Long duration, low depth (oscillating/modulated depth for Impressionist colors).
To compare these shapes while ignoring minor noise, the authors employed Fourier-descriptor analysis and a 5-point landmark analysis (comparing max, mean, and median values).
Experimental Setup
The framework was tested using the MAESTRO v3.0.0 dataset, which contains over 200 hours of virtuosic performances recorded on Disklavier pianos. This dataset provides perfectly synchronized MIDI and high-resolution optical sensor data for ground-truth pedal depth. Three Transformer-based models were evaluated: an audio-only binary model, an audio-only continuous model, and an AUDIO+MIDI model that incorporates note-velocity information as a structural prior.
Key Insights and Results
The study reveals that continuous depth estimation is essential; even for binary tasks, training on continuous data improves boundary detection and prevents the "abrupt" switching seen in binary-only models. Furthermore, the AUDIO+MIDI model emerged as the superior approach. While its frame-level MSE improvement was modest (0.0280 vs. 0.0416 for audio-only), it performed significantly better at the action and gesture levels.
As shown in the distribution analysis, the "Highland" gesture (long, deep holds) was the most accurately predicted across all models. However, the "Mountain" gesture remains a challenge, indicating that models still struggle with the fine micro-dynamics required for Impressionist-style pedaling. Notably, models trained on binary targets almost entirely failed to recognize complex patterns like "Mountain" or "Pinnacle," often collapsing their predictions into simplified "Highland" blocks.
Conclusion and Future Work
The proposed framework highlights the "MSE Trap," where models following musical patterns correctly are unfairly penalized by standard metrics. Despite the improvements, models still struggle with high max-depth ratios and very short, fast gestures that approach the model's temporal resolution limits.
Future research will focus on developing boundary-sensitive objectives and contour-aware loss functions to further bridge the gap between signal processing and musical perception. The authors also plan to validate these metrics through human perceptual studies.
Summary updated