HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription

Weixing Wei, Peilin Li, Yi Yu, Wei LiZenodo 2022 · DOI · OpenAlex

Abstract

While neural network models are making significant progress in piano transcription, they are becoming more resource-consuming due to requiring larger model size and more computing power. In this paper, we attempt to apply more prior about piano to reduce model size and improve the transcription performance. The sound of a piano note contains various overtones, and the pitch of a key does not change over time. To make full use of such latent information, we propose HPPNet, which uses Harmonic Dilated Convolution to capture harmonic structures and the Frequency Grouped Recurrent Neural Network to model pitch-invariance over time. Experimental results on the MAESTRO dataset show that our piano transcription system achieves state-of-the-art performance both in frame and note scores (frame F1 93.15%, note F1 97.18%). Moreover, the model size is much smaller than previous state-of-the-art deep learning models.

Introduction

Automatic Music Transcription (AMT) is a fundamental task in music information retrieval, aiming to convert raw audio into symbolic representations like MIDI. Piano transcription is particularly challenging due to its polyphonic nature, where multiple notes and their overlapping overtones occur simultaneously.

Recent trends in AMT have favored increasingly large models, such as Transformers or deep CNN-RNN architectures, which demand significant computational resources. However, these "black box" approaches often ignore the physical properties of the instrument. A piano note possesses a predictable harmonic structure (overtones) and a pitch that remains constant throughout its duration. HPPNet (Harmonic and Pitch-invariance Prior Network) addresses these inefficiencies by embedding these musical priors directly into the neural network architecture.

Methodology: Baking Physics into Architecture

HPPNet introduces two primary components designed to leverage domain knowledge:

Harmonic Dilated Convolution (HD-Conv)

Standard convolutional filters typically process adjacent pixels. In contrast, HD-Conv utilizes specific dilation rates designed to capture the relationship between a fundamental frequency ($f_0$) and its harmonics ($2f_0, 3f_0$, etc.). By using a Constant-Q Transform (CQT) input—where frequency is log-spaced—the distance between harmonics remains constant across the keyboard. This allows dilated convolutions to efficiently extract harmonic features without needing to learn these basic physical relationships from scratch.

Frequency Grouped LSTM (FG-LSTM)

Traditional models often flatten the frequency spectrum into a single vector, leading to massive parameter counts in the temporal modeling stage. HPPNet instead segments the acoustic features into 88 groups, corresponding to the 88 piano keys. These groups are processed by a shared, compact LSTM. This design models pitch invariance: the principle that the temporal evolution of a single note is largely independent of others. This approach reduces the temporal modeling parameters from millions to just 99k.

Conceptual overview of the HPPNet architecture, showing how HD-Conv captures harmonics and FG-RNN processes frequency groups.

Experimental Results

The model was evaluated on the MAESTRO (v1, v2, and v3) and MAPS datasets. HPPNet achieved state-of-the-art results while maintaining extreme parameter efficiency:

  • Accuracy: A Frame F1 score of 93.15% and a Note F1 score of 97.18% on MAESTRO v3, outperforming much larger models.
  • Efficiency: The model contains only 1.2 million parameters, roughly 45x fewer than leading Transformer-based models (54M parameters).
  • Robustness: HPPNet demonstrated better generalization on the MAPS dataset and higher performance when trained on limited data (e.g., 10% of MAESTRO), proving that built-in musical priors act as a powerful regularizer.
Validation F1 curves showing HPPNet (Proposed) outperforming ablation models lacking HD-Conv or FG-LSTM.

Conclusion and Future Work

HPPNet demonstrates that incorporating domain-specific priors can significantly improve both the accuracy and efficiency of deep learning models for music transcription. While highly effective for piano, future work includes optimizing the computational overhead of parallel dilated convolutions and adapting the architecture for instruments with variable pitch, such as the violin or human voice. Additionally, refining offset detection remains a priority for achieving even higher transcription fidelity.

Summary updated

Abstract

While neural network models are making significant progress in piano transcription, they are becoming more resource-consuming due to requiring larger model size and more computing power. In this paper, we attempt to apply more prior about piano to reduce model size and improve the transcription performance. The sound of a piano note contains various overtones, and the pitch of a key does not change over time. To make full use of such latent information, we propose HPPNet that using the Harmonic Dilated Convolution to capture the harmonic structures and the Frequency Grouped Recurrent Neural Network to model the pitch-invariance over time. Experimental results on the MAESTRO dataset show that our piano transcription system achieves state-of-the-art performance both in frame and note scores (frame F1 93.15%, note F1 97.18%). Moreover, the model size is much smaller than the previous state-of-the-art deep learning models.
BibTeX
@inproceedings{wei2022hppnet,
  author = {Weixing Wei and Peilin Li and Yi Yu and Wei Li},
  title = {{HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.7316762},
}

Loading related content...

Figure