Most foundational architectures in deep learning are designed for one-dimensional streams. Natural language processing treats text as a linear sequence of words, while audio models predict a single continuous waveform or a sequence of discrete tokens over time. When applied to music, this linear design naturally lends itself to monophonic generation—producing a single, isolated melodic line where only one note sounds at any given moment.
As a result, many generative frameworks operate under this monophonic restriction. For instance, Zhiqian Chen et al. note in their work on genre fusion, Learning to Fuse Music Genres with Generative Adversarial Dual Learning, that limiting generation to single-voiced melodies simplifies the modeling of temporal patterns. Similarly, systems that translate external modalities into music—such as converting human motion to soundtracks in How Does it Sound? by Kun Su et al., or translating speech prosody into melody in Musical Speech by Jason d'Eon et al.—frequently restrict their outputs to a single melodic instrument layered over a basic rhythm. Even in expressive performance modeling, Taizan Suzuki et al. point out in A Case Based Approach to the Generation of Musical Expression that models often focus exclusively on monophonic sequences, ignoring the complex interactions found in chords and polyphonic textures.
While a solo melody can be catchy, it lacks tonal context. As Marco Scirea et al. argue in their work on SMUG: Scientific Music Generator, generating isolated lines without an accompanying harmonic framework results in music that lacks emotional depth and structural integrity.
The Polyphonic Complexity Explosion
Why can't we easily feed chords or multi-instrument tracks into these models? The moment a system moves from monophony to polyphony, the underlying search space and structural complexity explode along two axes: horizontal (temporal) and vertical (harmonic).
In a polyphonic setting, a model cannot simply predict the next note based on past notes. It must coordinate multiple independent voices that move forward in time while ensuring they harmonize perfectly with one another at any given instant. Gaëtan Hadjeres, in developing DeepBach: a Steerable Model for Bach Chorales Generation, highlights that capturing these simultaneous vertical and horizontal dependencies is a massive challenge. If a model focuses too much on the horizontal melody of an individual voice, it risks creating harsh, unmusical dissonances vertically. If it focuses too much on vertical chord alignment, the individual lines lose their melodic momentum and sound robotic.
This scalability challenge applies to both symbolic music (like MIDI) and raw audio synthesis:
- Symbolic Constraints: In constraint-satisfaction and evolutionary approaches like EVOC: A Music Generating System using Genetic Algorithms, Timothy Weale et al. demonstrate that coordinating multiple independent lines while maintaining harmonic integrity dramatically expands the search space, posing severe scalability challenges for standard algorithms.
- Audio Synthesis Constraints: In differentiable digital signal processing (DDSP: Differentiable Digital Signal Processing), Jesse H. Engel et al. explain that frameworks relying on a single fundamental frequency ($f_0$) input are inherently bottlenecked. They cannot model the overlapping harmonics and complex resonance of polyphonic instruments like pianos or guitars, let alone full orchestral textures.
The Representational Bottleneck
The core reason standard sequence models fail at polyphony is a mismatch in data representation. As surveyed by Dinh-Viet-Toan Le et al. in Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey, music is inherently multi-dimensional. Standard language models designed for one-dimensional symbol streams struggle to capture simultaneous events without specialized adaptation.
When multiple notes occur at the same time step, representing relative relationships (like intervals and rhythmic offsets) becomes significantly more complex, as noted by Zixun Guo et al. in A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling. If a model flattens a multi-instrument arrangement into a single token stream, it loses the explicit boundaries between tracks. This makes it incredibly difficult to maintain melodic fidelity when dealing with dense textures, a limitation discussed by Shansong Liu et al. in Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer.
Historically, researchers have bypassed this by using hard-coded heuristic post-processing to add simultaneous notes to a generated melody. However, the authors of A-Muze-Net: Music Generation by Composing the Harmony based on the Generated Melody point out that relying on rule-based heuristics proves that the underlying sequence model has not natively learned polyphonic structures, limiting its ability to autonomously capture the complex interplay of concurrent musical events.
Strategies for Multi-Voice and Multi-Track Generation
To break through the monophonic barrier, recent research focuses on architectures that explicitly accommodate multi-dimensional structures, either through hierarchical modeling, tokenization strategies, or latent space disentanglement.
Hierarchical and Multi-Tiered Architectures
Instead of generating everything in a single flat sequence, models can separate the global harmonic structure from local melodic lines. For example, Zixun Guo et al. introduced Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure, a hierarchical neural network that generates melodies by explicitly conditioning them on underlying chord progressions, ensuring that the horizontal line respects the vertical harmony. In the audio domain, Yusong Wu et al. developed MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling, which splits synthesis into three distinct tiers (notes, performance, and synthesis) to provide granular control over expressive nuances while scaling past single-note limitations.
Latent Space Disentanglement and Multi-Track Diffusion
Another powerful approach involves encoding multi-track music into a shared latent space where instruments can be manipulated independently. Adam Roberts et al. demonstrated this in Learning a Latent Space of Multitrack Measures, allowing models to perform "attribute arithmetic" (such as adding or removing instruments) while maintaining polyphonic coherence across up to eight distinct tracks.
More recently, to avoid the limitation of generating a single, flattened audio mixture file—which prevents professional editing—Debottam Dutta et al. proposed Multi-Source Music Generation with Latent Diffusion. By encoding individual instrument tracks (stems) into a compressed, continuous representation using a shared VAE, they enable the simultaneous generation of distinct, editable instrumental layers that remain perfectly synchronized.
By moving away from rigid, one-dimensional sequences and embracing representations that natively understand simultaneous events, the field is steadily closing the polyphonic gap, transitioning AI music from simple solo melodies into rich, multi-instrumental compositions.
Key Foundational Work
A Case Based Approach to the Generation of Musical Expression (1999) — contributed 1 concept elements to this direction.
EVOC: A Music Generating System using Genetic Algorithms (2003) — contributed 1 concept elements to this direction.