TL;DR
Instruct-MusicGen is a lightweight, instruction-tuned version of the popular MusicGen model that enables precise text-based music editing, such as adding or removing specific instruments. By introducing just 8% more parameters and training for a mere 5,000 steps, it achieves state-of-the-art performance, outperforming models that were trained from scratch or rely on complex Large Language Model (LLM) reasoning.
What Problem Does This Paper Address?
While AI can now generate impressive music from text prompts, editing existing audio remains a significant hurdle. Professional music production relies on "stems"—individual tracks like drums or vocals—that can be isolated and manipulated. Current AI approaches to music editing generally fall into two camps, both with flaws:
- Training from scratch: These models are resource-intensive to build and often fail to match the audio quality of top-tier generative models.
- LLM-based "Reasoning": These systems use an LLM to interpret instructions and then ask a separate model to regenerate the music. This often leads to imprecise reconstruction where the "edited" version loses the character of the original audio.
The authors of Instruct-MusicGen sought a way to leverage the high-quality "knowledge" already embedded in pretrained music models while giving them the surgical precision needed for editing tasks.
Methods
Instead of building a new model, the researchers modified the existing MusicGen architecture using a strategy called "instruction tuning." They kept the core of the model frozen and added two specialized "fusion" modules:
- Audio Fusion Module: Inspired by LLaMA-Adapter, this module allows the model to "hear" the original audio. It uses a duplicated set of attention layers to extract features from the source music and integrates them into the main decoder using zero-initialized gating. This ensures that, at the start of training, the model's output is identical to the original, providing a stable foundation for learning edits.
- Text Fusion Module: This module uses LoRA (Low-Rank Adaptation) to tweak how the model processes text. Instead of just following a general description (e.g., "sad jazz"), it learns to follow specific commands (e.g., "Instruction: add drums").
By only training these small "adapter" layers, the team kept the number of trainable parameters low (264M) and the training time incredibly short (only 5,000 steps, or less than 1% of the time required to train a model from scratch).
Datasets
The model was trained and evaluated using two primary datasets:
- Slakh2100: A synthesized dataset derived from MIDI files. The authors created a custom "instruction" version of this dataset by programmatically adding, removing, or extracting stems to create ground-truth pairs for training.
- MoisesDB: A dataset of real-world audio tracks. This was used for "out-of-domain" evaluation to see if a model trained on synthesized MIDI sounds could generalize to complex, professional studio recordings.
Key Insights and Ideas
- The Power of Pretraining: The most striking finding is that you don't need massive datasets to teach a model how to edit. Because MusicGen already "understands" music structure, it only needs a tiny bit of extra guidance to learn how to manipulate it.
- Zero-initialized Gating is Crucial: By starting the gating factors at zero, the researchers ensured the model didn't "break" its existing generation capabilities during the first few steps of finetuning.
- Generalization across Domains: Despite being trained on the relatively "clean" Slakh dataset, the model performed remarkably well on the messy, complex audio of MoisesDB, suggesting it learned the concept of an instrument rather than just memorizing specific sounds.
Key Results
Instruct-MusicGen outperformed established baselines like AUDIT and M2UGen across almost every metric:
- Audio Quality: It achieved the lowest Fréchet Audio Distance (FAD), meaning its outputs sounded the most like real music.
- Instruction Adherence: In subjective listening tests, participants rated its ability to follow instructions at 3.85/5, compared to just 1.54/5 for AUDIT.
- Efficiency: It achieved these results with roughly 1/4th the trainable parameters and a fraction of the training data/time of its competitors.
Limitations and Future Work
While highly effective at "inter-stem" editing (adding/removing whole instruments), the paper notes that "intra-stem" editing—such as changing the style of a guitar solo without removing it—is a more complex challenge for future research. Additionally, the authors acknowledge that signal-level metrics like SI-SDR are often too harsh for generative models, as they penalize any slight deviation from the original waveform even if the result sounds perfect to a human ear. Future work may focus on more nuanced, creative controls and expanding the vocabulary of possible editing instructions.
Summary updated