Learning Disentangled Representations for Timber and Pitch in Music Audio

Yun-Ning Hung, Yi-An Chen, Yi-Hsuan YangarXiv 2018 · OpenReview

Abstract

Timbre and pitch are the two main perceptual properties of musical sounds. Depending on the target applications, we sometimes prefer to focus on one of them, while reducing the effect of the other. Researchers have managed to hand-craft such timbre-invariant or pitch-invariant features using domain knowledge and signal processing techniques, but it remains difficult to disentangle them in the resulting feature representations. Drawing upon state-of-the-art techniques in representation learning, we propose in this paper two deep convolutional neural network models for learning disentangled representation of musical timbre and pitch. Both models use encoders/decoders and adversarial training to learn music representations, but the second model additionally uses skip connections to deal with the pitch information. As music is an art of time, the two models are supervised by frame-level instrument and pitch labels using a new dataset collected from MuseScore. We compare the result of the two disentangling models with a new evaluation protocol called "timbre crossover", which leads to interesting applications in audio-domain music editing. Via various objective evaluations, we show that the second model can better change the instrumentation of a multi-instrument music piece without much affecting the pitch structure. By disentangling timbre and pitch, we envision that the model can contribute to generating more realistic music audio as well.

Introduction

In the domain of music information retrieval and synthesis, timbre (the unique quality of an instrument) and pitch (the frequency of the notes) are fundamental yet deeply intertwined properties. While humans effortlessly distinguish between a melody's structural content and its textural quality, computational models often struggle to separate these entangled features. Traditional approaches have relied on hand-crafted features to achieve invariance, but these methods often lack the flexibility required for advanced synthesis tasks. This paper addresses the challenge of learning disentangled representations, enabling applications such as "timbre crossover"—the ability to modify a recording's instrumentation while preserving its melodic integrity.

A key conceptual breakthrough in this work is the Boundary-Texture Analogy. The authors treat pitch in music similarly to object boundaries in images—it defines the "where" and "when"—while timbre provides the "surface" quality or texture. This analogy justifies the use of specific neural architectures to handle these distinct types of information.

Methods and Architectures

The core of this research involves two deep convolutional neural network (CNN) architectures designed to separate timbre and pitch information from raw audio input.

DuoAE and UnetAE

The first model, DuoAE (Dual Autoencoder), utilizes two distinct encoders to extract timbre and pitch codes. Disentanglement is enforced through adversarial training: a timbre classifier is trained to predict the instrument from the pitch code, while the main model is optimized to minimize the classifier's success, effectively "scrubbing" timbre information from the pitch representation.

The second model, UnetAE (U-Net Autoencoder), leverages the boundary-texture analogy. By employing a U-Net architecture with skip connections, the model passes high-resolution pitch information directly from the encoder to the decoder, while the bottleneck focuses on learning abstract timbre representations.

The UnetAE architecture uses skip connections to preserve pitch while the bottleneck focuses on timbre.

Dataset and Supervision

Training these models requires high-quality, time-aligned data. The authors compiled a dataset from MuseScore containing approximately 350,000 audio-MIDI pairs across nine instrument categories. Unlike models that reconstruct raw waveforms, these architectures are supervised by frame-level symbolic representations (pianorolls), ensuring that the latent spaces correspond to meaningful musical parameters.

Results and Evaluation

To evaluate the quality of disentanglement, the researchers introduced the "timbre crossover" protocol, where the timbre code of one piece is combined with the pitch information of another.

Quantitative and Qualitative Performance

The UnetAE model demonstrated superior performance, achieving a pitch transcription accuracy of 0.431, significantly higher than the DuoAE and baseline models. This suggests that the skip-connection architecture is more effective at preserving melodic structure. Furthermore, t-SNE visualizations of the learned timbre codes reveal that the model successfully clusters sounds by instrument, regardless of the notes being played. This provides strong evidence that the timbre representation is truly independent of pitch.

t-SNE visualization shows that timbre codes for different instruments (represented by colors) form distinct clusters.

Conclusion

This work demonstrates that deep learning architectures, particularly those incorporating skip connections and adversarial training, can effectively disentangle the complex relationship between timbre and pitch in music audio. While current limitations include the fidelity of STFT-based reconstruction and the lack of explicit pitch control in the latent space, the success of the "timbre crossover" application opens new avenues for creative music editing and high-quality audio synthesis.

Summary updated

Abstract

Timbre and pitch are the two main perceptual properties of musical sounds. Depending on the target applications, we sometimes prefer to focus on one of them, while reducing the effect of the other. Researchers have managed to hand-craft such timbre-invariant or pitch-invariant features using domain knowledge and signal processing techniques, but it remains difficult to disentangle them in the resulting feature representations. Drawing upon state-of-the-art techniques in representation learning, we propose in this paper two deep convolutional neural network models for learning disentangled representation of musical timbre and pitch. Both models use encoders/decoders and adversarial training to learn music representations, but the second model additionally uses skip connections to deal with the pitch information. As music is an art of time, the two models are supervised by frame-level instrument and pitch labels using a new dataset collected from MuseScore. We compare the result of the two disentangling models with a new evaluation protocol called "timbre crossover", which leads to interesting applications in audio-domain music editing. Via various objective evaluations, we show that the second model can better change the instrumentation of a multi-instrument music piece without much affecting the pitch structure. By disentangling timbre and pitch, we envision that the model can contribute to generating more realistic music audio as well.
BibTeX
@misc{hung2018learning,
  author = {Yun-Ning Hung and Yi-An Chen and Yi-Hsuan Yang},
  title = {{Learning Disentangled Representations for Timber and Pitch in Music Audio}},
  year = {2018},
  eprint = {1811.03271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03271},
}
Figure