Generating high-fidelity audio is a problem of extreme scale. To produce a single second of CD-quality music, a model must predict 44,100 individual samples. Yet, music is not just a sequence of samples; it is a nested hierarchy of structures. A single waveform contains micro-level timbral textures (milliseconds), meso-level rhythmic motifs and notes (seconds), and macro-level compositional forms like verses and choruses (minutes).
Standard sequence models often face a "resolution gap." If they focus on the fine-grained samples, they lose the "big picture" and wander aimlessly. If they focus on high-level structure, they lack the acoustic detail to sound realistic. To bridge this gap, researchers have developed hierarchical architectures that operate across multiple temporal scales simultaneously.
Bridging the Resolution Gap with Clock Rates
The most direct way to handle multiple scales is to build models with different layers that "tick" at different speeds. In SampleRNN: An Unconditional End-to-End Neural Audio Generation Model, Soroush Mehri and colleagues introduced a hierarchy of Recurrent Neural Networks (RNNs). The top-level RNN operates at a slow clock rate, capturing long-term structural trends, while the bottom-level RNN operates at the sample rate to ensure local smoothness.
This hierarchical approach was later scaled significantly in Jukebox: A Generative Model for Music. Prafulla Dhariwal and the OpenAI team used a multi-scale VQ-VAE to compress audio into discrete codes at different levels of abstraction. By training Transformers on these compressed codes, Jukebox can maintain musical coherence over several minutes while still generating raw audio that includes singing and complex instrumentation.
Multi-Window and Dilated Attention
While RNNs use different clock rates, Transformer-based models address the multi-scale problem through specialized attention mechanisms. A standard global attention mechanism treats every part of the signal with equal priority, which is computationally expensive and often fails to capture local nuances.
In Masked Autoencoders with Multi-Window Attention Are Better Audio Learners, Sarthak Yadav and colleagues proposed the Multi-Window Masked Autoencoder (MW-MAE). By allowing different attention "heads" to focus on varying window sizes—from highly local to fully global—the model can prioritize percussive transients and long-term melodic arcs at the same time. Similarly, Tae‐Jun Kim utilized Dilated Neighborhood Attention to analyze music, recognizing that low-level rhythmic features require short-term context while functional labels like "verse" or "chorus" require windows spanning several minutes.
Aligning Texture and Form
In symbolic music generation (like MIDI), the challenge shifts from waveforms to the relationship between "texture" (the vertical arrangement of notes in a chord) and "form" (the horizontal progression of phrases).
Xueyao Zhang and colleagues addressed this in Structure-Enhanced Pop Music Generation via Harmony-Aware Learning by using nested transformers. Their Harmony-Aware Hierarchical Music Transformer (HAT) explicitly groups local harmonic units into broader structural phrases. This ensures that the immediate accompaniment patterns remain consistent with the long-range compositional progression.
To further enforce this "part-whole" relationship, Dong Yao introduced Music-PAW, which uses a hierarchical contrastive loss. This training objective forces the model to align the latent representation of a short audio segment with the representation of the longer sequence containing it, teaching the model how local events contribute to the global semantic context.
Multi-Scale Synchronization
The need for multi-scale modeling is perhaps most evident in tasks requiring synchronization, such as generating music for a dance video or performing alongside a human. Kyle Olszewski and the creators of D2M-GAN used a dual-level structure to solve this: a high-level model captures the global musical structure, while a low-level model ensures that individual beats align perfectly with the dancer's movements.
This mirrors a "grand challenge" in the field described by Roger Dannenberg in A Vision of Creative Computation in Music Performance. For a computer to be a believable musical partner, it must synchronize across three distinct hierarchical levels: the beat, the measure, and the song section. As Dannenberg argues, a failure to align at any one of these scales breaks the illusion of a reactive, intelligent ensemble member.
By explicitly modeling these nested layers—whether through clock rates, multi-window attention, or hierarchical losses—modern research is moving away from treating audio as a flat stream of data and toward treating it as the complex, multi-scale architecture that it truly is.