Musical Composition Style Transfer via Disentangled Timbre Representations

Yun-Ning Hung, I-Tung Chiang, Yi-An Chen, Yi-Hsuan Yang

Abstract

Music creation involves both composition (melody, chords) and arrangement (instrument selection). While AI-driven composition has advanced significantly, automated arrangement remains less explored. This paper introduces deep learning models for rearranging polyphonic music across arbitrary genres. By employing adversarial training to disentangle latent factors of pitch (content) and timbre (instrumentation), our models can perform "composition style transfer"—rearranging a piece while preserving its original pitch content. We demonstrate the effectiveness of our approach through instrument activity detection and subjective listening tests.

Introduction

The art of music arrangement—deciding which instruments play which parts—is a critical yet complex component of music production. Traditional style transfer methods often operate on single-instrument audio or symbolic MIDI data. However, performing composition style transfer directly from polyphonic audio requires a model to simultaneously identify notes and instruments, separate them into independent representations, and recombine them coherently.

Data Representation and Problem Formulation

To facilitate disentanglement, the researchers decompose the standard musical representation into distinct components. The input is a Constant-Q Transform (CQT) of the audio, which provides a frequency-over-time representation aligned with human pitch perception. The target output is a multi-track "pianoroll."

Data representations: (a) Pianoroll, (b) Instrument roll, and (c) Pitch roll.

As shown in the figure above, a Pianoroll (a) can be viewed as a 3D tensor (time, frequency, instrument). This is decomposed into an Instrument roll (b), which captures which instruments are active over time regardless of pitch, and a Pitch roll (c), which captures the notes played regardless of which instrument plays them. The goal of the model is to learn latent codes that map specifically to these disentangled "Instrument" and "Pitch" features.

Proposed Architectures

The authors investigate two primary encoder-decoder architectures designed to learn these disentangled representations:

  1. DuoED (Dual Encoder-Decoder): This model uses two separate encoders—one for pitch and one for timbre. To ensure they don't "leak" information into each other, the authors use adversarial training.
  2. UnetED (U-Net Encoder-Decoder): Inspired by image segmentation, this model treats pitch as a structural feature and timbre as a textural feature.
The UnetED architecture featuring skip connections and auxiliary decoders for disentanglement.

In the UnetED model (illustrated above), the encoder $E_{cqt}$ extracts a latent timbre code $Z_t$. Crucially, skip connections pass low-level pitch information directly from the encoder to the decoder $D_{roll}$. To enforce disentanglement, auxiliary decoders ($D_t$ and $D_p$) are used during training to predict the instrument activity and pitch content respectively, ensuring the latent space $Z_t$ is optimized for timbre representation.

Experimental Results

The models were trained on the massive MuseScore Dataset (344,166 MIDI/MP3 pairs) and evaluated on real-world recordings from MedleyDB and Mixing Secret.

  • Instrument Activity Detection (IAD): UnetED achieved a state-of-the-art average AUC of 0.829 on the M&M dataset, proving its ability to accurately identify instruments in complex polyphonic mixes.
  • Style Transfer Quality: In user studies, UnetED's rearrangements were rated significantly higher for rhythm and harmony compared to baselines. Qualitatively, the model demonstrated "musical intelligence," such as assigning low-frequency notes to bass instruments when generating a band arrangement.

Conclusion

By disentangling pitch and timbre, this work enables a new form of music manipulation: composition style transfer. The UnetED architecture, leveraging the structural nature of pitch through skip connections, proves particularly effective at maintaining musical coherence during rearrangement. Future work will look toward expanding instrument variety and integrating vocal processing.

Summary updated

Abstract

Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best of our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a piece of polyphonic musical audio as input and predict as output its musical score. We investigate disentanglement techniques such as adversarial training to separate latent factors that are related to the musical content (pitch) of different parts of the piece, and that are related to the instrumentation (timbre) of the parts per short-time segment. By disentangling pitch and timbre, our models have an idea of how each piece was composed and arranged. Moreover, the models can realize "composition style transfer" by rearranging a musical piece without much affecting its pitch content. We validate the effectiveness of the models by experiments on instrument activity detection and composition style transfer. To facilitate follow-up research, we open source our code at https://github.com/biboamy/instrument-disentangle.
BibTeX
@inproceedings{hung2019musical,
  author = {Yun-Ning Hung and I-Tung Chiang and Yi-An Chen and Yi-Hsuan Yang},
  title = {{Musical Composition Style Transfer via Disentangled Timbre Representations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=bahP7hAiG9},
}

Loading related content...

Figure