Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling

Zhejing Hu, Yan Liu, Gong Chen, Bruce X. B. YuAAAI 2025 · DOI

Introduction

Recent advancements in AI music generation, exemplified by platforms like Suno and Udio, have demonstrated remarkable capabilities in creating full-length compositions. However, these models often function as "black boxes," providing users with limited granular control. A significant challenge in this domain is music infilling (or inpainting)—the process of modifying specific segments of a piece while maintaining the integrity of the surrounding context. Current symbolic music models typically treat this as a one-shot generation task, lacking the flexibility to incorporate iterative human feedback. When a model produces a suboptimal transition or a stylistically inconsistent phrase, users are frequently forced to regenerate the entire piece, leading to a frustrating creative experience.

To address these limitations, we present the Collaborative Music Inpainter (CMI), a human-in-the-loop framework designed to transform music infilling into a truly collaborative process between human and machine.

The CMI Framework

The CMI framework establishes a dynamic feedback loop where the user acts as an active participant in the generative process. Unlike traditional models that generate a single output, CMI allows users to mark specific segments as "unsatisfactory." This feedback is then used to refine the model's internal parameters in real-time, ensuring that subsequent generations align more closely with the user's intent.

The CMI workflow illustrates the iterative cycle between the JEP-AGA architecture, user evaluation, and the Dynamic Interaction Learner (DIL).

Technical Architecture

The core of the CMI system consists of two primary innovations: the Joint Embedding Predictive Autoregressive Generation Architecture (JEP-AGA) and the Dynamic Interaction Learner (DIL).

Joint Embedding Predictive Autoregressive Generation Architecture (JEP-AGA)

Inspired by Yann LeCun’s Joint Embedding Predictive Architecture (JEPA), the JEP-AGA moves beyond simple token-level prediction. Standard autoregressive models often suffer from "meaningless repetitions" because they lack a high-level understanding of musical structure. JEP-AGA addresses this by first learning latent representations of the missing (masked) segments based on the surrounding context. By predicting the "vibe" and structural essence of the piece before generating individual notes, the model ensures that the infilled content is both stylistically and harmonically consistent with the original work.

Dynamic Interaction Learner (DIL)

The DIL is the engine that enables real-time collaboration. It utilizes contrastive learning to update the model based on user responses after each interaction. The DIL operates through two distinct mechanisms:

  • Iteration-wise DIL: This component ensures that the model's next attempt is mathematically distant from the version the user just rejected, preventing the model from repeating the same mistakes.
  • Sample-wise DIL: This mechanism aligns the model's internal representations with the specific samples that the user finds acceptable, effectively personalizing the model to the user's unique aesthetic preferences during the session.
Detailed view of the JEP-AGA components (a) and the dual-path Dynamic Interaction Learner (b).

Experimental Evaluation

The CMI was rigorously evaluated using three major symbolic music (MIDI) datasets: Bread (851,313 files), LMD (176,581 files), and Pop909. The model was compared against state-of-the-art baselines, including XLNet and standard Transformer architectures.

Key Results

  • Objective Performance: CMI demonstrated statistically significant improvements in maintaining rhythmic, melodic, and harmonic consistency across large-scale datasets.
  • Subjective Quality: In a blind study involving 41 listeners, CMI outperformed baselines in key qualitative metrics, including Musicality (3.73/5), Naturalness (3.66/5), and Creativity (3.59/5).
  • Efficiency: The implementation of the DIL mechanism led to a 44% reduction in Interaction Cost (RIC) and a 20% increase in User Satisfaction (IUS), proving that the model can reach a desired output with significantly fewer human interventions.

Conclusion and Future Work

The Collaborative Music Inpainter represents a shift from AI as a solo creator to AI as a collaborative tool. By integrating high-level structural prediction with real-time human feedback, CMI provides a more intuitive and efficient workflow for musicians. Future research will focus on expanding this framework from symbolic MIDI data to raw audio waveforms and further refining the balance between model "fitness" and creative novelty. Additionally, the development of more sophisticated user interfaces will be essential to making these powerful collaborative tools accessible to a broader range of creators.

Summary updated

BibTeX
@inproceedings{hu2025compose,
  author = {Zhejing Hu and Yan Liu and Gong Chen and Bruce X. B. Yu},
  title = {{Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32122},
}
Figure