TL;DR
MEDIC introduces Disentangled Inversion Control (DIC), a training-free framework that enables precise zero-shot text-based music editing using pre-trained diffusion models. By splitting the generation process into a triple-branch setup—source, target, and an intermediate "harmonic" branch—it rectifies error accumulation from standard DDIM inversion while harmonizing cross-attention and mutual self-attention controls. Alongside the method, the authors present ZoME-Bench, the first dedicated 1,100-sample benchmark covering ten rigid and non-rigid music editing categories.
What Problem Does This Paper Address?
Zero-shot text-driven music editing allows users to modify existing audio tracks (e.g., changing instruments, genre, or mood) purely through textual instructions without retraining diffusion models. However, standard techniques face two fundamental bottlenecks:
- Inversion Drift & CFG Amplification: Typical DDIM inversion assumes the reverse ordinary differential equation (ODE) trajectory is completely reversible. In text-conditioned generation, this assumption fails. Introducing Classifier-Free Guidance (CFG) to enhance text adherence severely magnifies accumulated inversion errors, causing noticeable audio distortions and poor content preservation.
- The Rigid vs. Non-Rigid Dilemma: Existing attention manipulation mechanisms struggle with music. Cross-attention control (CAC) works well for rigid edits (e.g., adding/removing instruments), whereas mutual self-attention control works better for non-rigid edits (e.g., style, melody, or genre shifts). Naively chaining them together inside a standard dual-branch (source $\rightarrow$ target) architecture leads to structural conflict and degraded output quality.
Methods
MEDIC introduces a unified, training-free framework called Disentangled Inversion Control (DIC):
- Triple-Branch Architecture: Instead of the conventional two-branch scheme (source and target), DIC establishes three parallel paths:
- Source Branch: Generates the source track while applying a corrective offset at each step ($d^{\text{src}}_{t-1} = z^*_{t-1} - z^{\text{src}}_{t-1}$) to cancel out drift caused by DDIM inversion and CFG.
- Harmonic Branch: Acts as an intermediate structural bridge that hosts transformed melody and musical layout.
- Target Branch: Generates the final edited track without direct distance perturbations, ensuring maximum generative capacity and edit fidelity.
- Harmonized Attention Control (HAC):
- Mutual Self-Attention Control: Injected between the source and target latents to guide the intermediate harmonic branch, selectively applied in decoder layers after a designated denoising step threshold ($S$).
- Cross-Attention Control: Blends the attention maps of the intermediate harmonic branch and the target branch (via global attention refinement and localized token blending) before a temporal cutoff $\tau_c$.
[Source Branch] ---(Mutual Self-Attn)---> [Harmonic Branch] ---(Cross-Attn Control)---> [Target Branch]
(Drift Corrected) (Non-Rigid Edits) (Rigid Edits)
Datasets
- ZoME-Bench (Zero-shot Music Editing Benchmark): Created by the authors from MusicCaps audio (10-second clips at 16 kHz). It contains 1,100 curated samples partitioned across 10 editing tasks: change instrument, add instrument, delete instrument, change genre, change mood, change rhythm, change background, change melody, extract instrument, and random. Each entry includes source/target prompts, edit instructions, and blend words.
- MusicDelta (MedleyDB subset): 34 full-track musical excerpts ranging from 20 seconds to 5 minutes, used to evaluate variable-length music editing.
Key Insights and Ideas
- Decoupling Error Correction from Generation: Injecting trajectory corrections into the target branch constrains diffusion synthesis, hurting target-prompt adherence. By restricting distance corrections exclusively to the source branch, MEDIC maintains accurate structural references without limiting the target branch’s ability to generate new elements.
- The Intermediate "Harmonic Branch": Non-rigid musical transformations (like changing a melody or genre) alter the audio backbone, while rigid edits (adding an instrument) overlay new textures. Sequentially applying both attention controls inside two branches breaks attention refinement; routing non-rigid modifications into an intermediate harmonic branch resolves this structural conflict.
Key Results
Evaluated on AudioLDM 2 backbone:
- ZoME-Bench Benchmark: MEDIC outclassed both generative baselines (AudioLDM 2, MusicGen) and inversion baselines (DDIM Inversion, MusicMagus, DDPM-Friendly):
- Content Preservation: Lower Structure Distance ($11.97 \times 10^3$ vs. $16.23 \times 10^3$ for MusicMagus) and lower LPAPS ($0.15$ vs. $0.19$).
- Audio Quality & Fidelity: Best Frechet Audio Distance ($\text{FAD} = 2.49$ vs. $5.15$) and highest text-audio alignment ($\text{CLAP} = 0.61$ vs. $0.55$).
- Human Evaluation: Achieved the highest subjective ratings in both audio quality ($\text{MOS-Q} = 79.81$) and content preservation ($\text{MOS-P} = 77.29$).
- Variable-Length Editing (MusicDelta): Maintained state-of-the-art text alignment ($\text{CLAP} = 0.51$) where other inversion techniques suffered severe performance drops ($\le 0.43$).
Limitations and Future Work
- Computational Cost: Running 200 inference steps across three parallel branches requires approximately 92 seconds per sample on an NVIDIA A800/RTX 4090 GPU.
- Underlying Model Capacity: The scope and realism of edits remain bounded by the expressiveness of the underlying text-to-music backbone (e.g., AudioLDM 2), limiting drastic semantic restructurings.
- Future Directions: The authors aim to integrate fast consistency models or flow-matching frameworks to reduce latency and extend editing to more complex multi-attribute scenarios.
Summary updated