Abstract
Music creation involves not only composing parts like melodies and chords but also arranging instruments to play them. This paper presents the first deep learning models for rearranging music of arbitrary genres. By building encoders and decoders that take polyphonic audio and predict musical scores, the researchers investigate disentanglement techniques to separate musical content (pitch) from instrumentation (timbre). This allows for "composition style transfer"—rearranging a piece without affecting its pitch content.
Introduction: The Challenge of Music Rearrangement
In music production, "composing" (writing notes) and "arranging" (assigning instruments) are distinct creative acts. While AI has advanced in melody generation, automatic rearrangement—taking an existing piece and re-scoring it for a different ensemble—has remained difficult.
The primary technical hurdle is disentanglement: a model must understand "what" is being played independently of "what" is playing it. In standard audio, pitch and timbre are deeply intertwined, and there is a significant lack of paired data (the same song recorded with multiple arrangements) for supervised training. This research seeks a universal way to perform this "re-coloring" of music using only unpaired audio and MIDI data.
Methodology
The authors propose two architectures to separate pitch and timbre representations from a Constant-Q Transform (CQT) input, a spectrogram-like representation optimized for musical frequencies.
DuoED (Dual Encoder-Decoder)
The DuoED model utilizes two distinct encoders: one for timbre ($E_t$) and one for pitch ($E_p$). To prevent information leakage between these latent spaces, the authors employ adversarial training. Classifiers are trained to predict pitch from the timbre code and vice versa; the encoders are then optimized to "fool" these classifiers, ensuring the latent codes are truly disentangled.
UnetED (U-Net Encoder-Decoder)
Inspired by image segmentation, the UnetED model treats pitch as the "spatial structure" of the music and timbre as the "texture." Pitch information is preserved through high-resolution skip connections, while the bottleneck latent code ($Z_t$) is forced to represent only timbre. This architecture allows the model to maintain the structural integrity of the notes while swapping the instrumental "style."
Key Insights: Pitch as Structure, Timbre as Texture
A core insight of this work is the analogy between music and computer vision. By treating pitch as the "spatial" structure and timbre as the "style," the authors adapted U-Net architectures—typically used for medical imaging—to the musical domain. Furthermore, the models use temporal supervision, training on short-time segments to ensure the model tracks instrumental changes and note onsets precisely over time.
Evaluation and Results
The models were evaluated using the MuseScore dataset (synthetic MIDI/MP3 pairs) and the MedleyDB + Mixing Secret (M&M) dataset (real-world multi-track recordings).
- Instrument Activity Detection (IAD): The UnetED model achieved a state-of-the-art average Area Under the Curve (AUC) of 0.829 on real-world data, outperforming models without disentanglement.
- Composition Style Transfer: In subjective tests, humans rated UnetED rearrangements as more rhythmic and harmonic than baseline methods. Qualitatively, the model correctly assigned musical roles, such as placing low-pitched notes on the bass and high-pitched lines on lead instruments (e.g., flute or violin).
Conclusion
This framework enables the automatic "re-coloring" of music by successfully separating content from style. While the current output is a symbolic multi-track pianoroll requiring external synthesis, the researchers have demonstrated that adversarial disentanglement is a powerful tool for complex musical tasks. Future work aims to integrate neural audio synthesis for more realistic results and expand the instrument vocabulary beyond standard ensembles.
Summary updated