Abstract
AIRGen introduces a flexible framework for music editing that transforms standard autoregressive Large Language Models (LLMs) into powerful tools for inpainting, arrangement, and refinement. By utilizing a novel "heterogeneous adapter" and a specialized masking scheme, the framework enables fine-grained, content-based control over long-term music generation (filling gaps of 8+ seconds) while keeping the underlying massive model frozen.
Introduction
The rise of music LLMs like MusicGen has enabled high-quality generation, but these models are typically limited to unidirectional, "left-to-right" synthesis. This constraint hinders practical musical workflows such as iterative editing, where a musician might want to replace a specific segment, refine an existing track with new instrumentation, or arrange a composition based on a symbolic sketch.
Problem Statement
While Large Language Models are excellent at starting a song from scratch, they are inherently poor at iterative tasks. Existing music inpainting models are often limited to very short durations (under 2 seconds) and lack the ability to follow specific musical constraints like chord charts. AIRGen bridges this gap by repurposing the power of pretrained music LLMs for steerable, long-term editing.
Methodology
AIRGen builds upon MusicGen, a state-of-the-art transformer-based music model. Its primary innovation lies in the Heterogeneous Adapter combined with a unique tokenization design.
The Heterogeneous Adapter
Instead of a standard uniform adapter, AIRGen utilizes four distinct types of trainable adapters ($a_1$ through $a_4$). Each adapter is specialized for a specific "role" based on the frame's location:
- Prefix vs. Prediction: Whether the frame is part of the provided context or the area to be generated.
- Masked vs. Unmasked: Whether the frame is a target for synthesis or a provided condition (e.g., a drum track).
Tokenization and Input Design
The model uses EnCodec to convert audio into discrete tokens. To enable inpainting, a "prefix" is prepended to the input sequence containing both surrounding context (past and future audio) and specific conditions. Symbolic controls, such as MIDI piano covers or chord progressions, are algorithmically rendered into audio before being fed into the model to ensure input consistency.
Parameter-Efficient Fine-Tuning (PEFT)
The massive pretrained MusicGen model remains frozen. Only the lightweight adapters and gating factors are trained, making the process computationally efficient (requiring as few as 3.75M parameters) while preserving the original model's high-quality synthesis.
Experiments and Results
The framework was evaluated using the Slakh2100 dataset and the RWC-POP-100 dataset for out-of-distribution testing.
- Superior Inpainting: AIRGen significantly outperformed baselines like VampNet in long-gap inpainting tasks, achieving lower Frechet Audio Distance (FAD) scores.
- High Controllability: In drum-based refinement tasks, the model achieved a high Source-to-Distortion Ratio (SDR) of ~6.6, indicating tight alignment with provided tracks.
- Long-Term Stability: Unlike models that "drift" after a few seconds, AIRGen maintains semantic consistency over segments longer than 8 seconds by anchoring generation to both surrounding audio and content controls.
Conclusion
AIRGen demonstrates that autoregressive architectures are not a dead end for complex editing. By cleverly structuring the input prefix and employing heterogeneous adapters, the model can effectively "fill in the blanks" with high musical coherence and steerability. Future work will explore more diverse simultaneous controls and real-time interaction optimizations.
Summary updated