ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning

Daewoong Kim, Hao-Wen Dong, Dasaem Jeong

TL;DR

ViolinDiff is a two-stage diffusion-based framework that brings high-fidelity, expressive realism to polyphonic violin synthesis. By explicitly predicting and conditioning on pitch bend information via a novel "bend roll" representation, it captures subtle performance nuances like vibrato, significantly outperforming existing baseline models in both objective metrics and human listening tests.

What Problem Does This Paper Address?

While neural audio synthesis has achieved impressive realism in speech and singing (often reaching Mean Opinion Scores around 3.9), instrumental music synthesis has lagged behind (typically hovering around 3.1). A major bottleneck is polyphony.

In singing voice synthesis, fundamental frequency (F0) contours—which dictate pitch and expressiveness—are easily modeled as a simple 1D sequence because singing is predominantly monophonic. Instruments like the violin, however, frequently play multiple notes simultaneously. This polyphonic nature makes 1D F0 sequences impossible to use and complicates F0 estimation due to overlapping frequencies. Consequently, existing polyphonic models often omit explicit F0 modeling, leaving synthesized instruments sounding flat and lacking expressive nuances like vibrato.

Methods

The authors introduce ViolinDiff, a two-stage framework based on Denoising Diffusion Probabilistic Models (DDPM):

  • The "Bend Roll" Representation: To handle polyphony, the authors propose a new encoding scheme. Alongside standard binary piano rolls (which track frame activation, onset, and offset), they introduce a continuous bend roll ($R_{bend} \in (-1, 1)^{P \times T}$). This maps frame-wise pitch deviations directly onto a pitch-by-time grid, allowing the model to track independent pitch variations for simultaneous notes.
  • Two-Stage Architecture:
    1. Bend Estimation Module: Takes the binary piano rolls and predicts the continuous bend roll.
  • Synthesis Module: Takes both the binary piano rolls and the predicted bend roll to generate a highly detailed mel spectrogram.
  • Model Details: Both modules use a combination of Gated Recurrent Units (GRUs) and Transformer Encoders to extract note features. The denoiser is based on a non-causal WaveNet architecture. Feature-wise Linear Modulation (FiLM) layers are utilized to inject performer embeddings, allowing the model to capture unique stylistic and acoustic characteristics. Finally, a SoundStream vocoder converts the generated mel spectrogram into raw audio.
Overview of the ViolinDiff architecture showing the Bend Estimation Module and Synthesis Module.

Datasets

The model was trained and evaluated on a recently published solo violin dataset (Tamer et al., 2023). After filtering for quality, the dataset contained 961 audio recordings (approximately 34 hours) across 21 distinct performers. It features etudes by Wohlfahrt and Kayser, and caprices by Paganini. Crucially, the dataset includes synchronized MIDI files with detailed multi-pitch bend information obtained via automatic transcription and alignment.

Key Insights and Ideas

  • Polyphonic F0 as a Matrix: Instead of treating F0 as a 1D sequence, mapping pitch deviations to a 2D pitch-by-time "bend roll" elegantly solves the polyphonic representation problem.
  • Explicit vs. Implicit Modeling: Relying on a diffusion model to implicitly learn pitch trajectories from discrete MIDI notes results in a loss of fine-grained control. Explicitly separating pitch-bend prediction into a dedicated front-end module dramatically improves the clarity and naturalness of expressive micro-tonal variations.
  • Quantifying Vibrato: The authors introduce a novel evaluation metric for vibrato by applying a Discrete Fourier Transform (DFT) to the predicted F0 contours. This allows them to objectively measure both the presence of vibrato (F1 score) and stylistic accuracy (Perf-MAE) across different models.

Key Results

  • Objective Metrics: ViolinDiff consistently outperformed the baseline model (which lacked explicit pitch bend conditioning) across all Fréchet Audio Distance (FAD) variants and vibrato metrics, proving that explicit conditioning helps capture performer-specific and piece-specific qualities.
Quantitative performance comparison on FAD and Vibrato metrics.
  • Visualizing Vibrato: Spectrogram analysis shows that ViolinDiff generates significantly clearer and more distinct vibrato waveforms in polyphonic sections compared to the baseline.
Spectrogram comparison showing F0 contours (green) for original audio, ViolinDiff, and the baseline.
  • Subjective Evaluation: In a MUSHRA realism listening test, ViolinDiff achieved a mean score of 70.96, significantly outperforming the baseline (66.99) as well as prior state-of-the-art diffusion models by Maman et al. (59.47) and Hawthorne et al. (43.67).
Fig. 3. The box plot illustrates the results of the MUSHRA realism listening test. The p-values between models were calculated using the Wilcoxon signed-rank test. The green lines connect the compared models.

Limitations and Future Work

While ViolinDiff excels at pitch-based expressiveness, it currently lacks explicit controls for other vital performance parameters such as tempo dynamics and articulation (e.g., staccato vs. legato). The authors note that incorporating these control parameters is a key direction for future work. Additionally, they aim to extend this framework to synthesize other polyphonic instruments beyond the violin.

Summary updated

Abstract

Modeling the natural contour of fundamental frequency (F0) plays a critical role in music audio synthesis. However, transcribing and managing multiple F0 contours in polyphonic music is challenging, and explicit F0 contour modeling has not yet been explored for polyphonic instrumental synthesis. In this paper, we present ViolinDiff, a two-stage diffusion-based synthesis framework. For a given violin MIDI file, the first stage estimates the F0 contour as pitch bend information, and the second stage generates mel spectrogram incorporating these expressive details. The quantitative metrics and listening test results show that the proposed model generates more realistic violin sounds than the model without explicit pitch bend modeling. Audio samples are available online: daewoung.github.io/ViolinDiff-Demo.
BibTeX
@inproceedings{kim2025violindiff,
  author = {Daewoong Kim and Hao-Wen Dong and Dasaem Jeong},
  title = {{ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=91aO76z9Z7},
}
Figure