Abstract
Music creation involves both composition (melody, chords) and arrangement (instrument selection). While AI-driven composition has advanced significantly, automated arrangement remains less explored. This paper introduces deep learning models for rearranging polyphonic music across arbitrary genres. By employing adversarial training to disentangle latent factors of pitch (content) and timbre (instrumentation), our models can perform "composition style transfer"—rearranging a piece while preserving its original pitch content. We demonstrate the effectiveness of our approach through instrument activity detection and subjective listening tests.
Introduction
The art of music arrangement—deciding which instruments play which parts—is a critical yet complex component of music production. Traditional style transfer methods often operate on single-instrument audio or symbolic MIDI data. However, performing composition style transfer directly from polyphonic audio requires a model to simultaneously identify notes and instruments, separate them into independent representations, and recombine them coherently.
Data Representation and Problem Formulation
To facilitate disentanglement, the researchers decompose the standard musical representation into distinct components. The input is a Constant-Q Transform (CQT) of the audio, which provides a frequency-over-time representation aligned with human pitch perception. The target output is a multi-track "pianoroll."
As shown in the figure above, a Pianoroll (a) can be viewed as a 3D tensor (time, frequency, instrument). This is decomposed into an Instrument roll (b), which captures which instruments are active over time regardless of pitch, and a Pitch roll (c), which captures the notes played regardless of which instrument plays them. The goal of the model is to learn latent codes that map specifically to these disentangled "Instrument" and "Pitch" features.
Proposed Architectures
The authors investigate two primary encoder-decoder architectures designed to learn these disentangled representations:
- DuoED (Dual Encoder-Decoder): This model uses two separate encoders—one for pitch and one for timbre. To ensure they don't "leak" information into each other, the authors use adversarial training.
- UnetED (U-Net Encoder-Decoder): Inspired by image segmentation, this model treats pitch as a structural feature and timbre as a textural feature.
In the UnetED model (illustrated above), the encoder $E_{cqt}$ extracts a latent timbre code $Z_t$. Crucially, skip connections pass low-level pitch information directly from the encoder to the decoder $D_{roll}$. To enforce disentanglement, auxiliary decoders ($D_t$ and $D_p$) are used during training to predict the instrument activity and pitch content respectively, ensuring the latent space $Z_t$ is optimized for timbre representation.
Experimental Results
The models were trained on the massive MuseScore Dataset (344,166 MIDI/MP3 pairs) and evaluated on real-world recordings from MedleyDB and Mixing Secret.
- Instrument Activity Detection (IAD): UnetED achieved a state-of-the-art average AUC of 0.829 on the M&M dataset, proving its ability to accurately identify instruments in complex polyphonic mixes.
- Style Transfer Quality: In user studies, UnetED's rearrangements were rated significantly higher for rhythm and harmony compared to baselines. Qualitatively, the model demonstrated "musical intelligence," such as assigning low-frequency notes to bass instruments when generating a band arrangement.
Conclusion
By disentangling pitch and timbre, this work enables a new form of music manipulation: composition style transfer. The UnetED architecture, leveraging the structural nature of pitch through skip connections, proves particularly effective at maintaining musical coherence during rearrangement. Future work will look toward expanding instrument variety and integrating vocal processing.
Summary updated