Musical Composition Style Transfer via Disentangled Timbre Representations

Yun-Ning Hung, I-Tung Chiang, Yi-An Chen, Yi-Hsuan Yang

Abstract

Music creation involves not only composing parts like melodies and chords but also arranging instruments to play them. This paper presents the first deep learning models for rearranging music of arbitrary genres. By building encoders and decoders that take polyphonic audio and predict musical scores, the researchers investigate disentanglement techniques to separate musical content (pitch) from instrumentation (timbre). This allows for "composition style transfer"—rearranging a piece without affecting its pitch content.

Introduction: The Challenge of Music Rearrangement

In music production, "composing" (writing notes) and "arranging" (assigning instruments) are distinct creative acts. While AI has advanced in melody generation, automatic rearrangement—taking an existing piece and re-scoring it for a different ensemble—has remained difficult.

The primary technical hurdle is disentanglement: a model must understand "what" is being played independently of "what" is playing it. In standard audio, pitch and timbre are deeply intertwined, and there is a significant lack of paired data (the same song recorded with multiple arrangements) for supervised training. This research seeks a universal way to perform this "re-coloring" of music using only unpaired audio and MIDI data.

Methodology

The authors propose two architectures to separate pitch and timbre representations from a Constant-Q Transform (CQT) input, a spectrogram-like representation optimized for musical frequencies.

DuoED (Dual Encoder-Decoder)

The DuoED model utilizes two distinct encoders: one for timbre ($E_t$) and one for pitch ($E_p$). To prevent information leakage between these latent spaces, the authors employ adversarial training. Classifiers are trained to predict pitch from the timbre code and vice versa; the encoders are then optimized to "fool" these classifiers, ensuring the latent codes are truly disentangled.

The DuoED architecture uses adversarial training and dual encoders to separate pitch ($Z_p$) and timbre ($Z_t$) latent representations.

UnetED (U-Net Encoder-Decoder)

Inspired by image segmentation, the UnetED model treats pitch as the "spatial structure" of the music and timbre as the "texture." Pitch information is preserved through high-resolution skip connections, while the bottleneck latent code ($Z_t$) is forced to represent only timbre. This architecture allows the model to maintain the structural integrity of the notes while swapping the instrumental "style."

The UnetED architecture leverages skip connections to pass structural pitch information while bottlenecking timbre for style transfer.

Key Insights: Pitch as Structure, Timbre as Texture

A core insight of this work is the analogy between music and computer vision. By treating pitch as the "spatial" structure and timbre as the "style," the authors adapted U-Net architectures—typically used for medical imaging—to the musical domain. Furthermore, the models use temporal supervision, training on short-time segments to ensure the model tracks instrumental changes and note onsets precisely over time.

Evaluation and Results

The models were evaluated using the MuseScore dataset (synthetic MIDI/MP3 pairs) and the MedleyDB + Mixing Secret (M&M) dataset (real-world multi-track recordings).

  • Instrument Activity Detection (IAD): The UnetED model achieved a state-of-the-art average Area Under the Curve (AUC) of 0.829 on real-world data, outperforming models without disentanglement.
  • Composition Style Transfer: In subjective tests, humans rated UnetED rearrangements as more rhythmic and harmonic than baseline methods. Qualitatively, the model correctly assigned musical roles, such as placing low-pitched notes on the bass and high-pitched lines on lead instruments (e.g., flute or violin).

Conclusion

This framework enables the automatic "re-coloring" of music by successfully separating content from style. While the current output is a symbolic multi-track pianoroll requiring external synthesis, the researchers have demonstrated that adversarial disentanglement is a powerful tool for complex musical tasks. Future work aims to integrate neural audio synthesis for more realistic results and expand the instrument vocabulary beyond standard ensembles.

Summary updated

Abstract

Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best of our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a piece of polyphonic musical audio as input, and predict as output its musical score. We investigate disentanglement techniques such as adversarial training to separate latent factors that are related to the musical content (pitch) of different parts of the piece, and that are related to the instrumentation (timbre) of the parts per short-time segment. By disentangling pitch and timbre, our models have an idea of how each piece was composed and arranged. Moreover, the models can realize “composition style transfer” by rearranging a musical piece without much affecting its pitch content. We validate the effectiveness of the models by experiments on instrument activity detection and composition style transfer. To facilitate follow-up research, we open source our code at https://github.com/biboamy/instrument-disentangle.
BibTeX
@inproceedings{hung2019musical,
  author = {Yun-Ning Hung and I-Tung Chiang and Yi-An Chen and Yi-Hsuan Yang},
  title = {{Musical Composition Style Transfer via Disentangled Timbre Representations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=bahP7hAiG9},
}
Figure