Modeling Symbolic Music as Heterogeneous Graphs

19 concepts14 papers1 proposal

When humans listen to or compose music, we do not process it as a flat, linear stream of isolated sounds. Instead, our minds naturally organize music along multiple dimensions simultaneously: we follow individual melodic lines moving horizontally over time, feel the vertical harmony of chords blending concurrent notes, and recognize structural repetitions or themes occurring minutes apart.

Traditional deep learning architectures struggle to capture this multi-dimensional topology. Standard sequence models (like Transformers or RNNs) require music to be flattened into a 1-dimensional string of tokens, while grid-based models (like CNNs applied to piano rolls) treat music as a rigid 2-dimensional image. As Jeong et al. observe, these approaches introduce severe computational overhead or fail to resolve overlapping note durations and polyphonic relationships cleanly. Furthermore, without an explicit structural framework, sequential models often suffer from "aimlessness" Zou et al., losing track of the overarching theme and generating pieces that sound locally pleasant but structurally incoherent over longer spans Bhandari et al..

To bridge this gap, a prominent research direction models symbolic music as a heterogeneous (multi-relational) graph. By treating musical elements as nodes and their diverse interactions as typed edges, this framework provides a mathematically rigorous way to preserve the polyphonic, hierarchical, and long-term structures inherent to music.


The Anatomy of a Multi-Relational Music Graph

Instead of forcing music into a single chronological line, graph-based representations define musical primitives as nodes and connect them based on specific music-theoretic or temporal relationships. Depending on the task, these graphs operate at different levels of abstraction:

Note-Level Graphs

At the finest granularity, nodes represent individual notes, containing attributes like pitch, scale degree, and duration. Edges explicitly encode how these notes interact in time and space. For example, Karystinaios et al. construct graphs where edges define human perceptual principles, such as whether two notes share a simultaneous onset, have overlapping durations, or occur in immediate succession. To capture richer voice-leading dynamics, Ni-Hahn et al. utilize directed, heterogeneous edges categorized into types like forward, onset, sustain, and rest.

High-Level Structural Graphs

To capture macro-level organization, nodes can represent larger musical units. Cosenza et al. model multitrack music by treating chord activations as nodes, while Guo et al. use transposition-invariant interval sequences as nodes to discover unsupervised contrapuntal patterns. At an even larger scale, the MELONS framework proposed by Zou et al. organizes entire compositions by treating individual bars as nodes, linking them via eight distinct bar-level structural relations to guide long-term melody generation.

Note-Level Graph:       [Note: C4] --(simultaneous onset)--> [Note: E4]
                             |                                    |
                     (next in voice)                      (next in voice)
                             v                                    v
                        [Note: D4] --(duration overlap)----> [Note: F4]

Macro-Level Graph:      [ Bar 1 ]  --------(repetition)-------> [ Bar 5 ]

Why Graph Architectures Resolve Polyphony

The core intuition for why heterogeneous graphs outperform linear sequences in polyphonic settings lies in how Graph Neural Networks (GNNs) pass messages.

In a standard sequence model, tracking an individual melodic line (a voice) within a dense polyphonic arrangement requires the attention mechanism to "jump" over intervening tokens belonging to other concurrent voices or instruments. This dilutes the context window and mixes unrelated signals.

A multi-relational graph solves this by decoupling the horizontal (temporal) and vertical (harmonic) axes into separate edge types. As Cosenza et al. demonstrate in multitrack generation, edges can be explicitly drawn within a single track to capture melodic progression, and across different tracks to capture concurrent harmonic synchronization. When a GNN updates a node's embedding, it aggregates information along these distinct edge types independently. A note can thus update its representation based on its immediate melodic predecessor without interference from a bassline playing at the same time, while still remaining harmonically aware of that bassline through a separate "vertical" edge.

This explicit routing makes complex perceptual tasks highly tractable. For instance, Karystinaios et al. frame the difficult problem of separating interwoven musical voices as a link prediction task. Because the underlying graph natively maintains note-to-note temporal relationships, a heterogeneous GNN can learn to accurately predict whether a continuous voice-leading link exists between any two notes, effectively tracking multiple independent trajectories through a polyphonic score.


Encoding Hierarchical Theory and Long-Term Coherence

Music is inherently hierarchical; human listeners naturally distinguish between structural "pillar" notes that define a core harmony and transient "ornamental" notes (like passing tones or trills). Heterogeneous graphs allow computational models to operationalize these classical music-theoretical insights directly into their architecture.

Schenkerian Reductions and Node Pooling

Schenkerian analysis is a method of music theory that hierarchically reduces a composition down to its core structural backbone. The authors of AutoSchA Ni-Hahn et al. operationalize this by modeling scores as multi-relational graphs and introducing a "node isolation" pooling technique. By evaluating edge relationships like voice continuity and melodic intervals, the network systematically collapses ornamental note nodes while preserving structural ones, automatically generating a multi-layered hierarchical representation of the piece.

In generative contexts, such as the ProGress framework Ni-Hahn et al., combining discrete graph diffusion with these hierarchical Schenkerian connections ensures that newly generated surface-level notes remain strictly anchored to a globally coherent harmonic blueprint.

Melody Reduction as Shortest Path Finding

Framing music as a graph also unlocks classical algorithmic optimizations. Dannenberg et al. demonstrate that automatic melody reduction can be modeled as finding the shortest path through a directed note graph. By constructing "shortcut" edges that skip over intermediate notes, and weighting those edges using a cost function based on tonal stability and rhythmic importance, the system can mathematically identify the most structurally essential path through a melody.

Original Melody:   [C4] ----> [D4 (passing tone)] ----> [E4]
Graph Shortcut:    [C4] --------------------------------> [E4]  (Selected Reduction)

By bypassing the limitations of sequential context windows, heterogeneous graphs allow models to maintain global structural integrity. Whether by clustering thematic patterns across multi-minute spans Bhandari et al. or fusing multi-modal relationships between lyrics, chords, and song parts Wang et al., representing music as a relational network provides the structural scaffolding necessary for AI to understand and compose music with human-like coherence.

Key Foundational Work

Graph Neural Network for Music Score Data and Modeling Expressive Piano Performance (2019) — contributed 2 concept elements to this direction.
MELONS: generating melody with long-term structure using transformers and structure graph (2021) — contributed 1 concept elements to this direction.
Multi-Modal Chorus Recognition for Improving Song Search (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_graph_music_representation_670015dc,
  title = {heterogeneous graph music representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-graph-music-representation/rdr_670015dc44b04e96a7438ada49f13c8b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_graph_music_representation_670015dc_v1,
  title = {heterogeneous graph music representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-graph-music-representation/rdr_670015dc44b04e96a7438ada49f13c8b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake