Traditional machine learning treats music as a flat sequence—a string of MIDI tokens or a stream of audio frames. However, music is inherently hierarchical. A single note is part of a motif, which belongs to a phrase, which sits within a section, which defines the overall form of a movement. When models ignore this nested structure, they produce "musical word salad": sequences that sound fine locally but lack long-term logic, emotional arcs, or structural integrity.
To solve this, researchers are moving away from flat architectures toward models that explicitly represent the multi-layered "DNA" of musical composition.
The Flat Sequence Fallacy
The fundamental challenge in music AI is the gulf between a "perfect" technical playback and a coherent human performance. As Taizan Suzuki, Takenobu Tokunaga, and Hozumi Tanaka argued in A Case Based Approach to the Generation of Musical Expression, treating music as a flat sequence ignores the interactions between structural layers that inform expressive timing and dynamics.
If a model doesn't know where a phrase begins or ends, it cannot "breathe" like a human performer. This sentiment is echoed by Bill Z. Manaris and colleagues in A Corpus-Based Hybrid Approach to Music Analysis and Composition, who noted that generative systems failing to account for structural hierarchy inevitably produce music that lacks long-term coherence.
Mapping the Musical Skeleton
One way to capture this hierarchy is to treat music as a graph rather than a line. Ilana Shapiro, Hao-Wen Dong, and their team developed the Structural Temporal Graph (STG) in Deriving Representative Structure Over Music Corpora. The STG maps vertical relationships (like harmony) and horizontal progressions (like melody) into a single mathematical framework. By calculating a "centroid" graph, they can identify the shared architectural DNA of an entire genre.
A more radical approach involves Schenkerian analysis, a music theory method that strips away "decorative" notes to find the fundamental structural skeleton. Stephen Ni-Hahn and Cynthia Rudin introduced AutoSchA in AutoSchA: Automatic Hierarchical Music Representations via Multi-Relational Node Isolation, using a "node isolation" technique to automate this process. By distinguishing between primary harmonic notes and secondary ornaments, the model understands the "bones" of a piece, allowing for more sophisticated analysis and variation.
Top-Down Generation and Transcription
If music is hierarchical, then the process of creating or transcribing it should be too. Instead of predicting the next note, modern models often use hierarchical decoding.
In End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding, Wei Zeng and Ye Wang demonstrate a system that predicts bar-level features—like key and time signatures—before attempting to transcribe individual notes. This provides the necessary context to handle the complexity of human performance.
Similarly, for generation, Yikai Qian and Tianle Wang proposed MusicAOG in MusicAOG: an Energy-Based Model for Learning and Sampling a Hierarchical Representation of Symbolic Music. By treating composition as a structured "And-Or Graph" (a grammar-based approach), they achieve highly controllable generation that maintains structural logic across long durations, far outperforming massive but "flat" deep-learning models.
Learning the Part-Whole Relationship
To truly understand music, a model must recognize how small units aggregate into larger semantic structures. Dong Yao and Zhou Zhao addressed this in Music-PAW: Learning Music Representations via Hierarchical Part-whole Interaction and Contrast. Their "Part-Whole Transformer" uses hierarchical contrastive loss to explicitly model how phrases form passages.
This "mid-level" representation—what Tianle Wang calls "music-words" in Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music—bridges the gap between granular notes and dense audio. By discovering these recurring structural building blocks, AI can move beyond simple pattern matching toward what Nikhil Singh and Tod Machover call "musical common sense" in AI for Musical Discovery.
Why Hierarchy Works
The intuition behind this research direction is simple: constraints create coherence. By forcing a model to respect a hierarchy (bars $\rightarrow$ phrases $\rightarrow$ sections), we provide a scaffold that prevents the model from wandering aimlessly. Whether it is Ethan Manilow using hierarchies to improve instrument separation in Hierarchical Musical Instrument Separation, or Li Li using phrase-level attention for genre classification in Melodic Phrase Attention Network, the evidence is clear: when AI understands the layers of music, it understands the music itself.