When deep learning models process text, they treat it as a linear sequence of words. When early models attempted to process music the same way—as a flat, one-dimensional stream of notes or audio frames—they ran into a fundamental wall. Music is not a flat sequence. It is an inherently multi-layered, nested architecture where local events are tightly bound to global structures.
Treating music as a linear sequence forces a model to solve two incredibly difficult problems simultaneously: tracking low-level acoustic or symbolic mechanics (like pitch and duration) while maintaining long-term thematic coherence. Hierarchical music representation solves this by explicitly separating these layers, allowing models to reason about music the way humans compose and listen to it.
The Failure of Flat Sequences
In a flat sequence representation, every musical event is treated as an independent token in a single string. This design introduces severe computational and structural inefficiencies. As noted by Wu et al. in TunesFormer, processing music at a uniform note or character level forces high-level attention mechanisms to waste valuable capacity on low-level details. The model struggles to distinguish between the overarching thematic structure of a piece and the specific symbols inside a single measure.
Furthermore, musical events are multifaceted. A single moment in a score contains pitch, duration, meter, and velocity. Hsiao et al. point out in Compound Word Transformer that flattening these diverse attributes into a single sequence forces models to expend energy learning basic, coupled relationships that are inherently linked. Without structural boundaries, even powerful Large Language Models fail to manage complex musical relationships like motif development and structural form, as evaluated by Zhou et al. in Can LLMs "Reason" in Music?.
The Multi-Scale Architecture of Music
To capture musical logic, representations must account for two distinct hierarchical dimensions:
- The Horizontal (Temporal) Dimension: Notes group into motives, motives into multi-measure phrases, and phrases into large-scale sections like verses and choruses, as outlined by Bhandari et al. in Motifs, Phrases, and Beyond.
- The Vertical (Polyphonic) Dimension: Simultaneous notes form chords, and these chords must be coordinated across multiple parallel instrumental tracks to create a unified piece, as described by Cosenza et al. in Graph-based Polyphonic Multitrack Music Generation.
Global Score / Full Section
│
├─── Track 1 (Melody) ─── Phrase ─── Bar ─── Chord ─── Notes
│
└─── Track 2 (Harmony) ─── Phrase ─── Bar ─── Chord ─── Notes
A clear example of capturing this dual relationship is the PIANOTREE VAE framework developed by Wang et al.. Instead of a flat event stream, it uses a tree-structured architecture to organize polyphonic data into a nested sequence: individual notes form simultaneous note groups (chords), which then aggregate into global summaries.
By modeling chords as single units rather than independent notes, the system captures vertical relationships and horizontal temporal sequences simultaneously. This structural awareness ensures consistent expressive rendering within chords while drastically reducing the computational overhead of the generative process, a benefit also demonstrated by Rhyu et al. in Sketching the Expression.
Top-Down Constraints and Contextual Guidance
Why does a hierarchical representation make models perform better? The core intuition is that high-level musical properties constrain low-level choices. A note's validity or expressive timing is heavily dependent on its position within a bar, a chord progression, or a global key signature.
In End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding, Zeng et al. demonstrate that a hierarchical decoder predicting bar-level features (like key and time signatures) before generating note-level sequences significantly improves transcription accuracy. The global context acts as a corrective lens: knowing the bar-level constraints allows the model to utilize global context to correct local note-level errors.
This top-down guidance is equally vital for generation. Eck et al. show in A Probabilistic Model for Chord Progressions that a binary tree graphical model can represent the hierarchical structure of musical meter, where internal nodes capture the relationships between strong and weak beats. This architecture anchors local transitions within a larger rhythmic framework, ensuring global coherence.
Without this anchoring, flat models suffer from structural drift. As shown by Dai et al. in What is missing in deep music generation?, the statistical properties of low-level elements like chords are fundamentally influenced by their position within higher-level structures like phrases; failing to manage these multi-level dependencies leads to a loss of intentional repetition and structural logic.
Structural Reduction and Long-Term Coherence
Hierarchical modeling also enables structural reduction, a concept deeply rooted in music theory. Instead of treating every note as equally important, music can be viewed as a fundamental background skeleton decorated by surface variations.
Ni-Hahn et al. implement this computationally in AutoSchA by treating symbolic music scores as multi-relational graphs and using a node isolation pooling technique to automate structural decomposition across voices and temporal scales. Similarly, Carnovalini et al. show in A New Corpus for Computational Music Research andA Novel Method for Musical Structure Analysis that melodies can be reduced into hierarchical trees by iteratively identifying the most significant notes based on harmonic context and metric position. The root of the tree represents the fundamental core of a musical phrase, making it possible to quantify how surface variations deviate from a shared melodic skeleton.
When models operate on these semantically grouped, reduced representations—such as the measure-based "bar patching" used by Wu et al. in MelodyT5—sequence complexity drops dramatically. In music structure analysis, the author of Learning Multi-Level Representations for Hierarchical Music Structure Analysis. utilizes multi-level triplet sampling to create disentangled embeddings where different sections of a data vector capture musical structures at different time scales simultaneously.
By organizing the latent space around a nested hierarchy of sections, phrases, and motifs, generative frameworks like the one proposed by Qian et al. in MusicAOG maintain long-range musical logic. The model no longer has to blindly predict the next note; it follows a structural roadmap where high-level arrangement decisions govern low-level execution.