TL;DR
This paper introduces PiRhDy, a novel framework for learning rich, multi-faceted embeddings of symbolic music. By simultaneously considering pitch, rhythm, and dynamics, and modeling both melodic and harmonic contexts, PiRhDy significantly advances the state-of-the-art in representing musical information for downstream tasks.
What Problem Does This Paper Address?
Representing symbolic music (like MIDI data) in a way that deep learning models can effectively use is a persistent challenge. Unlike natural language, where words are relatively discrete, musical "tokens" (like individual notes) are complex, carrying information about pitch, duration (rhythm), and loudness (dynamics). Furthermore, music has a two-dimensional context: a horizontal melodic flow and a vertical harmonic structure. Existing methods often focus on only one or two of these aspects, leading to incomplete representations. This paper aims to bridge this gap by creating a unified framework that captures these crucial musical dimensions.
Methods
PiRhDy employs a two-stage hierarchical approach:
- Token Modeling Network: This stage focuses on individual musical events (note events). It first extracts key features:
- Pitch: Combines chroma (pitch class) and octave.
- Rhythm: Uses inter-onset interval (IOI) and note state (on, hold, off).
-
Dynamics: Captures velocity (loudness). These features are then embedded into dense vectors. Finally, a fusion module integrates these embeddings into a single, unified representation for each note event.
-
Context Modeling Network: This stage learns from the relationships between these note event embeddings. It uses a hierarchical strategy:
- Local Context: A sliding window approach, inspired by word2vec, predicts a center note event from its melodic and harmonic neighbors. This involves RNNs and attention mechanisms to capture temporal dependencies.
- Global Context: This stage fine-tunes the embeddings using sequence-level (period-level) and track-level (song-level) contexts, allowing the model to learn longer-term musical structures.
Datasets
- Lakh MIDI Dataset: Used for training and evaluating the PiRhDy framework. This dataset is a large collection of MIDI files.
- TOP-MAGD and MASD datasets: Used for genre classification, a song-level downstream task.
Key Insights and Ideas
- Music is inherently multi-faceted: Treating musical notes as simple tokens is insufficient. Capturing pitch, rhythm, and dynamics simultaneously is crucial for effective representation.
- Two-dimensional context matters: Both the horizontal flow (melody) and the vertical structure (harmony) are vital for understanding music. PiRhDy's ability to model both is a key differentiator.
- Hierarchical learning is effective: Learning from individual note events (token-level) and then building up to phrases and entire songs (sequence/song-level) allows for a more comprehensive understanding of musical structure.
- Melody often dominates: While harmonic context is important, the study suggests that melodic continuity plays a more significant role in certain tasks like melody completion.
- Feature-based vocabulary is more robust: Representing musical features (pitch, rhythm, dynamics) separately and then combining their embeddings proved more effective and efficient than using a large, Cartesian product-based vocabulary.
Key Results
- PiRhDy embeddings significantly outperform baseline models (like melody2vec, tonnetz, and pianoroll) on three downstream tasks: melody completion, accompaniment assignment, and genre classification.
- In melody completion, PiRhDy achieved near-perfect scores (e.g., HITS@10 of 0.971 for PiRhDy_GM).
- For accompaniment assignment, PiRhDy also showed strong performance, demonstrating its ability to capture vertical musical relationships.
- In genre classification, PiRhDy_GM (melody-preferred embeddings) achieved the highest AUC-ROC and F1 scores on both TOP-MAGD and MASD datasets, indicating its effectiveness for song-level analysis.
Limitations and Future Work
- IOI's limited contribution: The study found that integrating the Inter-Onset Interval (IOI) feature provided less improvement than other features, possibly due to limited variation in the dataset or the nature of IOI in representing temporal relationships.
- Melody vs. Harmony importance: While PiRhDy models both, the results suggest that melodic context is often more influential for certain tasks. Future work could explore more nuanced ways to balance these contexts.
- Future Directions: The authors propose extending PiRhDy to incorporate rhythm and tonal patterns more explicitly, analogous to syntax in natural language. They also suggest exploring multimodal approaches by integrating audio content and applying PiRhDy to a broader range of tasks like music similarity matching and generation.
Summary updated