ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning

Daewoong Kim, Hao-Wen Dong, Dasaem Jeong

Abstract

Modeling the natural contour of fundamental frequency (F0) plays a critical role in music audio synthesis. However, transcribing and managing multiple F0 contours in polyphonic music is challenging, and explicit F0 contour modeling has not yet been explored for polyphonic instrumental synthesis. In this paper, we present ViolinDiff, a two-stage diffusion-based synthesis framework. For a given violin MIDI file, the first stage estimates the F0 contour as pitch bend information, and the second stage generates mel spectrogram incorporating these expressive details. The quantitative metrics and listening test results show that the proposed model generates more realistic violin sounds than the model without explicit pitch bend modeling. Audio samples are available online: daewoung.github.io/ViolinDiff-Demo.

Introduction

Musical instruments, particularly the violin, present a significant challenge for neural audio synthesis due to the intricate nuances of human performance, such as vibrato and portamento. While singing voice synthesis (SVS) has achieved high realism by leveraging fundamental frequency (F0) contours, applying similar techniques to polyphonic instruments is difficult. A single F0 curve cannot represent multiple simultaneous notes, and standard MIDI files often lack the micro-variations in pitch that characterize expressive playing. ViolinDiff addresses this "quality gap" by explicitly modeling pitch deviations through a two-stage diffusion framework.

Methodology

The ViolinDiff architecture utilizes Denoising Diffusion Probabilistic Models (DDPM) across two specialized modules. The first stage, the Bend Estimation Module, predicts a "Bend Roll" from a standard MIDI score. This Bend Roll serves as a continuous representation of pitch deviation across all active notes, effectively acting as a proxy for F0 in a polyphonic context. The second stage, the Synthesis Module, incorporates this predicted Bend Roll along with MIDI onsets, offsets, and a Performer Embedding to generate a mel spectrogram. Finally, a pre-trained SoundStream vocoder converts the mel spectrogram into the final audio waveform.

The two-stage ViolinDiff architecture: (a) Bend Estimation Module and (b) Synthesis Module.

Technical innovations include the use of Feature-wise Linear Modulation (FiLM) layers to condition the denoising process on performer-specific styles and a hybrid architecture combining GRU/Transformer blocks with a non-causal WaveNet-based denoiser. The model was trained on a high-quality dataset comprising 34 hours of solo violin performances from 21 distinct performers, featuring detailed multi-pitch bend information transcribed from real recordings.

Results and Discussion

Evaluation of ViolinDiff demonstrates significant improvements over baselines that lack explicit pitch bend modeling. Quantitatively, the model achieved a vibrato F1 score of 0.567, compared to 0.408 for the "NoBend" baseline, indicating superior capture of pitch oscillations. It also recorded lower Fréchet Audio Distance (FAD) scores across all categories (All-FAD, Performer-FAD, and Piece-FAD), suggesting higher perceptual similarity to ground-truth recordings.

Realism Mean Scores from subjective listening tests, comparing ViolinDiff to various baselines and reference audio.

Subjective MUSHRA listening tests further validated these findings, with ViolinDiff earning a mean realism score of 70.96. This score is notably higher than the baseline (66.99) and previous state-of-the-art diffusion models such as Maman et al. (59.47). These results highlight that making pitch bend an explicit intermediate step allows the synthesis module to focus on generating high-fidelity timbre that follows realistic pitch movement.

Conclusion

ViolinDiff successfully bridges the gap between robotic MIDI synthesis and expressive human performance by explicitly conditioning on pitch bends. While the current framework focuses on pitch, future work could expand to include controls for tempo and articulation, or adapt the model for other expressive instruments like the cello. Despite the computational cost associated with iterative diffusion processes, ViolinDiff represents a major step toward high-fidelity, expressive instrumental synthesis.

Summary updated

Abstract

Modeling the natural contour of fundamental frequency (F0) plays a critical role in music audio synthesis. However, transcribing and managing multiple F0 contours in polyphonic music is challenging, and explicit F0 contour modeling has not yet been explored for polyphonic instrumental synthesis. In this paper, we present ViolinDiff, a two-stage diffusion-based synthesis framework. For a given violin MIDI file, the first stage estimates the F0 contour as pitch bend information, and the second stage generates mel spectrogram incorporating these expressive details. The quantitative metrics and listening test results show that the proposed model generates more realistic violin sounds than the model without explicit pitch bend modeling. Audio samples are available online: daewoung.github.io/ViolinDiff-Demo.
BibTeX
@inproceedings{kim2025violindiff,
  author = {Daewoong Kim and Hao-Wen Dong and Dasaem Jeong},
  title = {{ViolinDiff: Enhancing Expressive Violin Synthesis with Pitch Bend Conditioning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=91aO76z9Z7},
}
Figure