Abstract
The Segmented Full-Song Model (SFS) is a novel framework for symbolic piano music generation designed to mimic the human compositional process. By factorizing a song into discrete segments and utilizing a Transformer architecture that selectively attends to structurally relevant "anchor" sections, SFS generates coherent full-length songs significantly faster than previous diffusion-based models. This approach enables real-time human-AI co-creation by supporting non-linear generation workflows and maintaining long-range thematic consistency.
Introduction
Generating full-length musical compositions (3–5 minutes) remains a significant challenge for symbolic AI. Standard autoregressive models often suffer from "context drift," where the model loses structural coherence over long sequences as early information is pushed out of the attention window. While hierarchical diffusion models have been proposed to mitigate this, they are often computationally prohibitive and fail to distinguish between generic context and the specific thematic "motifs" that define a song's identity. Furthermore, traditional models generate music linearly, which contradicts the iterative, non-linear process used by human composers who might start with a catchy hook and build the structure around it.
Methods
The Segment-Factorized Full-Song Generation (SFS) framework addresses these issues through three core innovations: selective contextual attention, a global vision module, and flexible generation ordering.
Selective Contextual Attention
Rather than attending to all previous tokens uniformly, SFS identifies four "anchor" context types for any given segment to maintain structural integrity:
- Left/Right Neighbors: Ensures smooth transitions between adjacent segments.
- Seed: The primary musical theme or motif of the song.
- Reference (Ref): A previous segment with the same structural role (e.g., Chorus 1 serves as a reference for Chorus 2).
Model Architecture
The system consists of two primary components: the Generator (P) and the Global Vision Module (G).
- Global Vision Module (G): This module uses a pretrained VAE to encode segments into compact, bar-level embeddings. This provides a coarse summary of the entire song's structure, preventing the model from losing the "big picture" without overwhelming the Transformer with token-level detail.
- Generator (P): A Transformer decoder that integrates the global context and selective local context to produce note tokens. It is trained on all possible permutations of segment orders, enabling flexible, non-chronological generation.
Tokenization
Music is represented using a frame-based approach with 1/8 beat resolution. Notes are decomposed into sub-tokens representing pitch, velocity, and duration, allowing for a more granular and expressive representation of piano performances.
Datasets
The model was trained on an in-house dataset of 32,090 pop piano performances, with 3,615 performances reserved for testing. To simplify the learning space, all songs were transposed to C major or A minor. Segmentation was performed using a custom rule-based similarity algorithm.
Key Results
SFS demonstrates significant improvements in both efficiency and quality over state-of-the-art baselines:
- Inference Speed: SFS achieves an inference speed of 2.03 beats/second, a 10x improvement over the diffusion-based WholeSong baseline (0.197 beats/second).
- Thematic Consistency: In user evaluations, SFS scored 3.59/5 for "Adherence to Seed," significantly outperforming WholeSong (3.16/5) and flat GPT-style baselines (2.34/5).
- Long-Range Structure: Objective metrics (Structureness Indicator) confirm that SFS maintains patterns across 16+ bars more effectively than existing models.
Conclusion
SFS represents a shift from brute-force context processing to a structured, factorized approach that aligns with human creativity. By prioritizing structurally relevant context and providing a global view of the composition, the model achieves high thematic consistency and operational efficiency. While future work is needed to improve phrase-level transitions and incorporate semantic labels, SFS provides a robust foundation for real-time, co-creative AI music systems.
Summary updated