Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning

Yixiao Zhang, Yukara Ikemiya, Woosung Choi, Naoki Murata, Marco A. Martínez Ramírez, Liwei Lin, Gus Xia, Wei-Hsiang Liao, Yuki Mitsufuji, Simon Dixon

Abstract

Recent advances in text-to-music editing, which employ text queries to modify music (e.g., by changing its style or adjusting instrumental components), present unique challenges and opportunities for AI-assisted music creation. Previous approaches in this domain have been constrained by the necessity to train specific editing models from scratch, which is both resource-intensive and inefficient; other research uses large language models to predict edited music, resulting in imprecise audio reconstruction. To combine the strengths and address these limitations, we introduce Instruct-MusicGen, a novel approach that finetunes a pretrained MusicGen model to efficiently follow editing instructions such as adding, removing, or separating stems. Our approach involves a modification of the original MusicGen architecture by incorporating a text fusion module and an audio fusion module, which allow the model to process instruction texts and audio inputs concurrently and yield the desired edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters to the original MusicGen model and only trains for 5K steps, yet it achieves superior performance across all tasks compared to existing baselines, and demonstrates performance comparable to the models trained for specific tasks. This advancement not only enhances the efficiency of text-to-music editing but also broadens the applicability of music language models in dynamic music production environments.

Introduction

While text-to-music generation has seen significant progress, text-to-music editing—the ability to modify existing audio via natural language commands—remains a complex challenge. Traditional methods often require training specialized models from scratch or rely on Large Language Models (LLMs) that "hallucinate" new tracks, failing to preserve the identity of the original source music.

Instruct-MusicGen addresses these issues by transforming the popular MusicGen model into a precise music editor through instruction tuning. By leveraging a pretrained generative model, the framework learns to follow commands like "add drums" or "remove guitar" while maintaining the core melody and rhythm of the input audio.

Comparison between standard MusicGen (generation) and Instruct-MusicGen (editing).

Methodology

The core of Instruct-MusicGen lies in its efficient adaptation of the MusicGen architecture. The researchers introduced two lightweight fusion modules that enable the model to process both source audio and editing instructions simultaneously.

Audio Fusion Module

The Audio Fusion module incorporates features from the source audio (encoded via EnCodec) into the MusicGen decoder. A critical component is the zero-initialized gating factor. By starting the gating factor at zero, the model begins training in a stable state, initially just reproducing the input before gradually learning how to apply modifications.

Text Fusion Module

To distinguish between general descriptions and specific editing commands, the model utilizes Low-Rank Adaptation (LoRA) on the cross-attention layers of the T5 text encoder. This allows the model to interpret instructions like "Instruction: add drums" effectively without retraining the entire text encoder.

The architecture of Instruct-MusicGen featuring Text and Audio Fusion modules.

Experimental Setup

The model was trained and evaluated on two key datasets:

  • Slakh2100: A dataset of 2,100 MIDI-synthesized tracks, providing ground-truth triplets of {input, instruction, output} for tasks like stem addition and removal.
  • MoisesDB: A dataset of 240 real-world tracks used for out-of-domain evaluation to test the model's robustness on professional recordings.

Results

Instruct-MusicGen demonstrates state-of-the-art performance with minimal overhead, adding only 8% more parameters and requiring just 5,000 training steps on a single A100 GPU.

  • Objective Performance: On the Slakh dataset, the model achieved a Fréchet Audio Distance (FAD) of 3.75, significantly outperforming the AUDIT baseline (6.88).
  • Signal Integrity: In instrument removal tasks, it achieved a Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) of -2.09, a massive improvement over the -45.60 seen in baseline models, indicating superior preservation of the original audio.
  • Subjective Evaluation: Human listeners rated Instruct-MusicGen higher in both instruction adherence (3.85/5) and audio quality (3.55/5) compared to existing baselines.

Conclusion

Instruct-MusicGen proves that powerful music language models can be efficiently adapted for editing tasks without extensive retraining. While currently focused on inter-stem editing (adding/removing instruments) in short clips, future work aims to expand capabilities to intra-stem editing (timbre transfer) and longer audio durations. This research marks a significant step toward intuitive, AI-assisted music production.

Summary updated

Abstract

Recent advances in text-to-music editing, which employ text queries to modify music (e.g.\ by changing its style or adjusting instrumental components), present unique challenges and opportunities for AI-assisted music creation. Previous approaches in this domain have been constrained by the necessity to train specific editing models from scratch, which is both resource-intensive and inefficient; other research uses large language models to predict edited music, resulting in imprecise audio reconstruction. To Combine the strengths and address these limitations, we introduce Instruct-MusicGen, a novel approach that finetunes a pretrained MusicGen model to efficiently follow editing instructions such as adding, removing, or separating stems. Our approach involves a modification of the original MusicGen architecture by incorporating a text fusion module and an audio fusion module, which allow the model to process instruction texts and audio inputs concurrently and yield the desired edited music. Remarkably, Instruct-MusicGen only introduces 8% new parameters to the original MusicGen model and only trains for 5K steps, yet it achieves superior performance across all tasks compared to existing baselines, and demonstrates performance comparable to the models trained for specific tasks. This advancement not only enhances the efficiency of text-to-music editing but also broadens the applicability of music language models in dynamic music production environments.
BibTeX
@inproceedings{zhang2025instruct,
  author = {Yixiao Zhang and Yukara Ikemiya and Woosung Choi and Naoki Murata and Marco A. Martínez Ramírez and Liwei Lin and Gus Xia and Wei-Hsiang Liao and Yuki Mitsufuji and Simon Dixon},
  title = {{Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning}},
  booktitle = {ISMIR},
  year = {2025},
  doi = {10.5281/zenodo.17811375},
}
Figure