MEDIC: Zero-shot Music Editing with Disentangled Inversion Control

Huadai Liu, Jialei Wang, Xiangtai Li, Rongjie Huang, Yang Liu, Jiayang Xu, Zhou ZhaoSubmission to ICLR 2025 · Withdrawn

TL;DR

MEDIC introduces Disentangled Inversion Control (DIC), a training-free framework that enables precise zero-shot text-based music editing using pre-trained diffusion models. By splitting the generation process into a triple-branch setup—source, target, and an intermediate "harmonic" branch—it rectifies error accumulation from standard DDIM inversion while harmonizing cross-attention and mutual self-attention controls. Alongside the method, the authors present ZoME-Bench, the first dedicated 1,100-sample benchmark covering ten rigid and non-rigid music editing categories.

What Problem Does This Paper Address?

Zero-shot text-driven music editing allows users to modify existing audio tracks (e.g., changing instruments, genre, or mood) purely through textual instructions without retraining diffusion models. However, standard techniques face two fundamental bottlenecks:

  1. Inversion Drift & CFG Amplification: Typical DDIM inversion assumes the reverse ordinary differential equation (ODE) trajectory is completely reversible. In text-conditioned generation, this assumption fails. Introducing Classifier-Free Guidance (CFG) to enhance text adherence severely magnifies accumulated inversion errors, causing noticeable audio distortions and poor content preservation.
  2. The Rigid vs. Non-Rigid Dilemma: Existing attention manipulation mechanisms struggle with music. Cross-attention control (CAC) works well for rigid edits (e.g., adding/removing instruments), whereas mutual self-attention control works better for non-rigid edits (e.g., style, melody, or genre shifts). Naively chaining them together inside a standard dual-branch (source $\rightarrow$ target) architecture leads to structural conflict and degraded output quality.

Methods

MEDIC introduces a unified, training-free framework called Disentangled Inversion Control (DIC):

  • Triple-Branch Architecture: Instead of the conventional two-branch scheme (source and target), DIC establishes three parallel paths:
  • Source Branch: Generates the source track while applying a corrective offset at each step ($d^{\text{src}}_{t-1} = z^*_{t-1} - z^{\text{src}}_{t-1}$) to cancel out drift caused by DDIM inversion and CFG.
  • Harmonic Branch: Acts as an intermediate structural bridge that hosts transformed melody and musical layout.
  • Target Branch: Generates the final edited track without direct distance perturbations, ensuring maximum generative capacity and edit fidelity.
  • Harmonized Attention Control (HAC):
  • Mutual Self-Attention Control: Injected between the source and target latents to guide the intermediate harmonic branch, selectively applied in decoder layers after a designated denoising step threshold ($S$).
  • Cross-Attention Control: Blends the attention maps of the intermediate harmonic branch and the target branch (via global attention refinement and localized token blending) before a temporal cutoff $\tau_c$.
[Source Branch]  ---(Mutual Self-Attn)--->  [Harmonic Branch]  ---(Cross-Attn Control)--->  [Target Branch]
   (Drift Corrected)                           (Non-Rigid Edits)                                (Rigid Edits)

Datasets

  • ZoME-Bench (Zero-shot Music Editing Benchmark): Created by the authors from MusicCaps audio (10-second clips at 16 kHz). It contains 1,100 curated samples partitioned across 10 editing tasks: change instrument, add instrument, delete instrument, change genre, change mood, change rhythm, change background, change melody, extract instrument, and random. Each entry includes source/target prompts, edit instructions, and blend words.
  • MusicDelta (MedleyDB subset): 34 full-track musical excerpts ranging from 20 seconds to 5 minutes, used to evaluate variable-length music editing.

Key Insights and Ideas

  • Decoupling Error Correction from Generation: Injecting trajectory corrections into the target branch constrains diffusion synthesis, hurting target-prompt adherence. By restricting distance corrections exclusively to the source branch, MEDIC maintains accurate structural references without limiting the target branch’s ability to generate new elements.
  • The Intermediate "Harmonic Branch": Non-rigid musical transformations (like changing a melody or genre) alter the audio backbone, while rigid edits (adding an instrument) overlay new textures. Sequentially applying both attention controls inside two branches breaks attention refinement; routing non-rigid modifications into an intermediate harmonic branch resolves this structural conflict.

Key Results

Evaluated on AudioLDM 2 backbone:

  • ZoME-Bench Benchmark: MEDIC outclassed both generative baselines (AudioLDM 2, MusicGen) and inversion baselines (DDIM Inversion, MusicMagus, DDPM-Friendly):
  • Content Preservation: Lower Structure Distance ($11.97 \times 10^3$ vs. $16.23 \times 10^3$ for MusicMagus) and lower LPAPS ($0.15$ vs. $0.19$).
  • Audio Quality & Fidelity: Best Frechet Audio Distance ($\text{FAD} = 2.49$ vs. $5.15$) and highest text-audio alignment ($\text{CLAP} = 0.61$ vs. $0.55$).
  • Human Evaluation: Achieved the highest subjective ratings in both audio quality ($\text{MOS-Q} = 79.81$) and content preservation ($\text{MOS-P} = 77.29$).
  • Variable-Length Editing (MusicDelta): Maintained state-of-the-art text alignment ($\text{CLAP} = 0.51$) where other inversion techniques suffered severe performance drops ($\le 0.43$).

Limitations and Future Work

  • Computational Cost: Running 200 inference steps across three parallel branches requires approximately 92 seconds per sample on an NVIDIA A800/RTX 4090 GPU.
  • Underlying Model Capacity: The scope and realism of edits remain bounded by the expressiveness of the underlying text-to-music backbone (e.g., AudioLDM 2), limiting drastic semantic restructurings.
  • Future Directions: The authors aim to integrate fast consistency models or flow-matching frameworks to reduce latency and extend editing to more complex multi-attribute scenarios.

Summary updated

Abstract

Text-guided diffusion models make a paradigm shift in audio generation, facilitating the adaptability of source audio to conform to specific textual prompts. Recent works introduce inversion techniques, like DDIM inversion, to zero-shot editing, exploiting pretrained diffusion models for audio modification. Nonetheless, our investigation exposes that DDIM inversion suffers from an accumulation of errors across each diffusion step, undermining its efficacy. Moreover, existing editing methods fail to achieve effective complex non-rigid music editing while maintaining essential content preservation and high editing fidelity. To counteract these issues, we introduce the Disentangled Inversion technique to disentangle the diffusion process into triple branches, rectifying the deviated path of the source branch caused by DDIM inversion. In addition, we propose the Harmonized Attention Control framework, which unifies the mutual self-attention control and cross-attention control with an intermediate Harmonic Branch to progressively achieve the desired harmonic and melodic information in the target music. Collectively, these innovations comprise the Disentangled Inversion Control (DIC) framework, enabling accurate music editing while safeguarding content integrity. To benchmark audio editing efficacy, we introduce ZoME-Bench, a comprehensive music editing benchmark hosting 1,100 samples spread across ten distinct editing categories. This facilitates both zero-shot and instruction-based music editing tasks. Our method achieves unparalleled performance in edit fidelity and essential content preservation, outperforming contemporary state-of-the-art inversion techniques. Audio samples are available at https://MEDIC-Zero.github.io. Both code and benchmark will be released.
BibTeX
@misc{liu2024medic,
  author = {Huadai Liu and Jialei Wang and Xiangtai Li and Rongjie Huang and Yang Liu and Jiayang Xu and Zhou Zhao},
  title = {{MEDIC: Zero-shot Music Editing with Disentangled Inversion Control}},
  year = {2024},
  eprint = {2407.13220},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.13220},
}
Figure