Abstract
Automatic Piano Transcription (AMT) involves converting raw audio recordings into symbolic music representations like MIDI. This paper introduces the hFT-Transformer, a novel hierarchical architecture designed to address the complexities of polyphonic piano music. By processing spectral and temporal information in two distinct stages—a frequency hierarchy followed by a time hierarchy—the model effectively captures harmonic structures and long-term temporal dependencies. Experimental results on the MAPS and MAESTRO datasets demonstrate that the hFT-Transformer achieves state-of-the-art performance across multiple metrics, including frame, note, and velocity estimation.
Introduction
Transcribing piano music is a fundamental challenge in music information retrieval due to high polyphony and overlapping harmonic content. Successful AMT requires simultaneous modeling of harmonic relationships (frequency axis) and temporal evolution (time axis). While Convolutional Neural Networks (CNNs) and Transformers have been applied to this task, existing methods often suffer from resolution loss during feature aggregation or struggle with long-range temporal dependencies. The hFT-Transformer addresses these limitations through a hierarchical approach that preserves fine-grained spectral detail before analyzing temporal patterns.
Methodology: The hFT-Transformer Architecture
The core of the hFT-Transformer is its two-level hierarchy, which processes a log-mel spectrogram input sequentially to decouple pitch detection from temporal dynamics.
1. Frequency Hierarchy
The first stage focuses on the vertical frequency axis to resolve harmonic structures.
- Preprocessing: A 1D-CNN block processes the time axis locally, acting as a pre-emphasis filter without prematurely aggregating spectral information.
- Transformer Encoder (Frequency): This component analyzes relationships between frequency bins within individual time frames, identifying the fundamental frequencies and overtones that constitute a note.
- The Converter (Transformer Decoder): A key innovation is the use of a Transformer Decoder with cross-attention to map spectral features to the 88 piano keys. This "query-based" approach allows the model to precisely identify pitches from complex spectral data.
2. Time Hierarchy
The output from the frequency hierarchy is transposed and passed to the second stage. This Transformer Encoder (Time axis) captures the long-term evolution of notes, which is essential for accurate offset detection (when a note ends) and velocity estimation (how hard a note was struck).
Inference Strategy: The Half-Stride Trick
During inference, the model processes audio in discrete chunks. The researchers observed that transcription errors are not uniformly distributed; rather, they tend to increase at the boundaries of these chunks due to limited context.
To mitigate this, a half-stride strategy is employed. By overlapping chunks by 50% and retaining only the central, most accurate portion of each prediction, the model significantly improves the stability and accuracy of the final transcription across long recordings.
Experimental Results
The hFT-Transformer was evaluated on the MAPS and MAESTRO v3.0.0 datasets, establishing new state-of-the-art benchmarks:
- MAESTRO: The model achieved a Note F1-score of 97.44% and a Note with Offset and Velocity F1-score of 89.48%.
- MAPS: It outperformed previous heavyweights like Onsets&Frames, particularly in Note with Offset estimation (66.34% F1).
- Ablation Studies: The researchers found that 1D-CNNs outperformed 2D-CNNs, suggesting that preserving spectral resolution is more beneficial than early 2D feature aggregation. Additionally, treating velocity as a 128-category classification problem using cross-entropy proved superior to standard regression-based approaches.
Conclusion and Future Work
The hFT-Transformer demonstrates the efficacy of hierarchical processing for complex audio tasks. By prioritizing frequency resolution before temporal modeling, it achieves unprecedented accuracy in piano transcription. Future research will focus on adapting the "Converter" for continuous-pitch instruments and extending the architecture to multi-instrument orchestral settings.
Summary updated