PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music

Hongru Liang, Wenqiang Lei, Paul Yaozhu Chan, Zhenglu Yang, Maosong Sun, Tat-Seng Chua

TL;DR

This paper introduces PiRhDy, a novel framework for learning rich, multi-faceted embeddings of symbolic music. By simultaneously considering pitch, rhythm, and dynamics, and modeling both melodic and harmonic contexts, PiRhDy significantly advances the state-of-the-art in representing musical information for downstream tasks.

What Problem Does This Paper Address?

Representing symbolic music (like MIDI data) in a way that deep learning models can effectively use is a persistent challenge. Unlike natural language, where words are relatively discrete, musical "tokens" (like individual notes) are complex, carrying information about pitch, duration (rhythm), and loudness (dynamics). Furthermore, music has a two-dimensional context: a horizontal melodic flow and a vertical harmonic structure. Existing methods often focus on only one or two of these aspects, leading to incomplete representations. This paper aims to bridge this gap by creating a unified framework that captures these crucial musical dimensions.

Methods

PiRhDy employs a two-stage hierarchical approach:

  1. Token Modeling Network: This stage focuses on individual musical events (note events). It first extracts key features:
  2. Pitch: Combines chroma (pitch class) and octave.
  3. Rhythm: Uses inter-onset interval (IOI) and note state (on, hold, off).
  4. Dynamics: Captures velocity (loudness). These features are then embedded into dense vectors. Finally, a fusion module integrates these embeddings into a single, unified representation for each note event.

  5. Context Modeling Network: This stage learns from the relationships between these note event embeddings. It uses a hierarchical strategy:

  6. Local Context: A sliding window approach, inspired by word2vec, predicts a center note event from its melodic and harmonic neighbors. This involves RNNs and attention mechanisms to capture temporal dependencies.
  7. Global Context: This stage fine-tunes the embeddings using sequence-level (period-level) and track-level (song-level) contexts, allowing the model to learn longer-term musical structures.

Datasets

  • Lakh MIDI Dataset: Used for training and evaluating the PiRhDy framework. This dataset is a large collection of MIDI files.
  • TOP-MAGD and MASD datasets: Used for genre classification, a song-level downstream task.

Key Insights and Ideas

  • Music is inherently multi-faceted: Treating musical notes as simple tokens is insufficient. Capturing pitch, rhythm, and dynamics simultaneously is crucial for effective representation.
  • Two-dimensional context matters: Both the horizontal flow (melody) and the vertical structure (harmony) are vital for understanding music. PiRhDy's ability to model both is a key differentiator.
  • Hierarchical learning is effective: Learning from individual note events (token-level) and then building up to phrases and entire songs (sequence/song-level) allows for a more comprehensive understanding of musical structure.
  • Melody often dominates: While harmonic context is important, the study suggests that melodic continuity plays a more significant role in certain tasks like melody completion.
  • Feature-based vocabulary is more robust: Representing musical features (pitch, rhythm, dynamics) separately and then combining their embeddings proved more effective and efficient than using a large, Cartesian product-based vocabulary.

Key Results

  • PiRhDy embeddings significantly outperform baseline models (like melody2vec, tonnetz, and pianoroll) on three downstream tasks: melody completion, accompaniment assignment, and genre classification.
  • In melody completion, PiRhDy achieved near-perfect scores (e.g., HITS@10 of 0.971 for PiRhDy_GM).
  • For accompaniment assignment, PiRhDy also showed strong performance, demonstrating its ability to capture vertical musical relationships.
  • In genre classification, PiRhDy_GM (melody-preferred embeddings) achieved the highest AUC-ROC and F1 scores on both TOP-MAGD and MASD datasets, indicating its effectiveness for song-level analysis.

Limitations and Future Work

  • IOI's limited contribution: The study found that integrating the Inter-Onset Interval (IOI) feature provided less improvement than other features, possibly due to limited variation in the dataset or the nature of IOI in representing temporal relationships.
  • Melody vs. Harmony importance: While PiRhDy models both, the results suggest that melodic context is often more influential for certain tasks. Future work could explore more nuanced ways to balance these contexts.
  • Future Directions: The authors propose extending PiRhDy to incorporate rhythm and tonal patterns more explicitly, analogous to syntax in natural language. They also suggest exploring multimodal approaches by integrating audio content and applying PiRhDy to a broader range of tasks like music similarity matching and generation.

Summary updated

Abstract

Definitive embeddings remain a fundamental challenge of computational musicology for symbolic music in deep learning today. Analogous to natural language, music can be modeled as a sequence of tokens. This motivates the majority of existing solutions to explore the utilization of word embedding models to build music embeddings. However, music differs from natural languages in two key aspects: (1) musical token is multi-faceted – it comprises of pitch, rhythm and dynamics information; and (2) musical context is two-dimensional – each musical token is dependent on both melodic and harmonic contexts. In this work, we provide a comprehensive solution by proposing a novel framework named PiRhDy that integrates pitch, rhythm, and dynamics information seamlessly. PiRhDy adopts a hierarchical strategy which can be decomposed into two steps: (1) token (i.e., note event) modeling, which separately represents pitch, rhythm, and dynamics and integrates them into a single token embedding; and (2) context modeling, which utilizes melodic and harmonic knowledge to train the token embedding. A thorough study was made on each component and sub-strategy of PiRhDy. We further validate our embeddings in three downstream tasks – melody completion, accompaniment suggestion, and genre classification. Results indicate a significant advancement of the neural approach towards symbolic music as well as PiRhDy’s potential as a pretrained tool for a broad range of symbolic music applications.
BibTeX
@inproceedings{liang2020pirhdy,
  author = {Hongru Liang and Wenqiang Lei and Paul Yaozhu Chan and Zhenglu Yang and Maosong Sun and Tat-Seng Chua},
  title = {{PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=l18B8iLxZh},
}
Figure