Music Style Transfer: A Position Paper

Shuqi Dai, Zheng Zhang, Gus XiaarXiv 2018 · DOI · arXiv

Abstract

Led by the success of neural style transfer on visual arts, there has been a rising trend very recently in the effort of music style transfer. However, "music style" is not yet a well-defined concept from a scientific point of view. The difficulty lies in the intrinsic multi-level and multi-modal character of music representation (which is very different from image representation). As a result, depending on their interpretation of "music style", current studies under the category of "music style transfer", are actually solving completely different problems that belong to a variety of sub-fields of Computer Music. Also, a vanilla end-to-end approach, which aims at dealing with all levels of music representation at once by directly adopting the method of image style transfer, leads to poor results. Thus, we vitally propose a more scientifically-viable definition of music style transfer by breaking it down into precise concepts of timbre style transfer, performance style transfer and composition style transfer, as well as to connect different aspects of music style transfer with existing well-established sub-fields of computer music studies. In addition, we discuss the current limitations of music style modeling and its future directions by drawing spirit from some deep generative models, especially the ones using unsupervised learning and disentanglement techniques.

Introduction: The Definition Problem

In the visual arts, "style transfer"—the process of rendering the content of one image in the style of another (e.g., turning a photograph into a Van Gogh painting)—has achieved remarkable success. This success has naturally spurred researchers to attempt similar feats in the domain of music. However, music presents a unique challenge: it is fundamentally multi-level and multi-modal.

Unlike an image, which is a static grid of pixels, music exists simultaneously as an abstract score (the "what"), a performance (the "how"), and a physical sound wave (the "texture"). Because "style" can refer to any of these layers, the field has become fragmented. Researchers often solve completely different problems under the same "music style transfer" label, leading to a confusing "many-to-one" collapse of terminology.

A spectrogram visualizing the frequency content of audio over time, representing the timbre level of music style.

Furthermore, "vanilla" approaches that directly apply image-based neural networks to audio spectrograms often fail. These models tend to produce "casual remixes" or noisy textures rather than meaningful stylistic transformations, as they ignore the long-term temporal structures and complex grammars inherent to music.

A Scientific Taxonomy of Music Style

To address this ambiguity, this paper proposes a formal scientific taxonomy. Rather than treating music style as a monolithic concept, we break it down into three distinct, manageable categories:

  1. Timbre Style Transfer (Sound Level): This involves altering the sound texture or "color" while preserving the underlying performance. An example would be making a trumpet melody sound as if it were played on a flute.
  2. Performance Style Transfer (Control Level): This focuses on altering expressive nuances—such as timing, dynamics, and articulation—while preserving the original notes. For instance, taking a MIDI file of a classical piece and applying the "swing" and rhythmic displacement of a jazz solo.
  3. Composition Style Transfer (Score Level): This level preserves the core melodic identity while altering the harmony, rhythm, or arrangement. A classic example is re-composing a simple nursery rhyme in the style of a Bach Chorale.

The Abstraction/Realization Paradox

A key insight of this framework is the definition of "content" versus "style" through the lens of abstraction. Content can be viewed as the information extracted when moving from sound to score (the process of abstraction). Conversely, Style is the information added when moving from score to sound (the process of realization or interpretation).

The difficulty in current AI models lies in disentanglement. To achieve true style transfer, a model must learn latent representations that perfectly separate the invariant "content" from the variable "style" at each level. Most current models suffer from "structural blindness," struggling with the high-level architecture of music, such as phrases, repetitions, and long-term tension.

Current State and Future Directions

As of current research, progress is uneven across these three categories. Timbre transfer has seen early success with models like Google’s WaveNet, though long-form audio remains difficult. Composition transfer has succeeded with monophonic melodies, but polyphonic arrangements often lack the "hidden grammar" required to sound human.

Notably, there is a distinct "Bach Bias" in existing literature. Most successful composition models focus on J.S. Bach, likely because his musical grammar is highly rule-based and "local," making it a more accessible target for current AI than composers with more global or dynamic structures. Performance transfer remains a nascent field with significant room for growth.

The ultimate goal for the field is a cross-modal, end-to-end system. Such a system would be capable of taking raw audio, understanding its score and performance nuances, and re-synthesizing it in a completely different style across all three levels—timbre, performance, and composition—simultaneously.

Summary updated

Abstract

Led by the success of neural style transfer on visual arts, there has been a rising trend very recently in the effort of music style transfer. However, "music style" is not yet a well-defined concept from a scientific point of view. The difficulty lies in the intrinsic multi-level and multi-modal character of music representation (which is very different from image representation). As a result, depending on their interpretation of "music style", current studies under the category of "music style transfer", are actually solving completely different problems that belong to a variety of sub-fields of Computer Music. Also, a vanilla end-to-end approach, which aims at dealing with all levels of music representation at once by directly adopting the method of image style transfer, leads to poor results. Thus, we vitally propose a more scientifically-viable definition of music style transfer by breaking it down into precise concepts of timbre style transfer, performance style transfer and composition style transfer, as well as to connect different aspects of music style transfer with existing well-established sub-fields of computer music studies. In addition, we discuss the current limitations of music style modeling and its future directions by drawing spirit from some deep generative models, especially the ones using unsupervised learning and disentanglement techniques.
BibTeX
@misc{dai2018music,
  author = {Shuqi Dai and Zheng Zhang and Gus Xia},
  title = {{Music Style Transfer: A Position Paper}},
  year = {2018},
  eprint = {1803.06841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.06841},
}
Figure