TL;DR
ViolinDiff is a two-stage diffusion-based framework that brings high-fidelity, expressive realism to polyphonic violin synthesis. By explicitly predicting and conditioning on pitch bend information via a novel "bend roll" representation, it captures subtle performance nuances like vibrato, significantly outperforming existing baseline models in both objective metrics and human listening tests.
What Problem Does This Paper Address?
While neural audio synthesis has achieved impressive realism in speech and singing (often reaching Mean Opinion Scores around 3.9), instrumental music synthesis has lagged behind (typically hovering around 3.1). A major bottleneck is polyphony.
In singing voice synthesis, fundamental frequency (F0) contours—which dictate pitch and expressiveness—are easily modeled as a simple 1D sequence because singing is predominantly monophonic. Instruments like the violin, however, frequently play multiple notes simultaneously. This polyphonic nature makes 1D F0 sequences impossible to use and complicates F0 estimation due to overlapping frequencies. Consequently, existing polyphonic models often omit explicit F0 modeling, leaving synthesized instruments sounding flat and lacking expressive nuances like vibrato.
Methods
The authors introduce ViolinDiff, a two-stage framework based on Denoising Diffusion Probabilistic Models (DDPM):
- The "Bend Roll" Representation: To handle polyphony, the authors propose a new encoding scheme. Alongside standard binary piano rolls (which track frame activation, onset, and offset), they introduce a continuous bend roll ($R_{bend} \in (-1, 1)^{P \times T}$). This maps frame-wise pitch deviations directly onto a pitch-by-time grid, allowing the model to track independent pitch variations for simultaneous notes.
- Two-Stage Architecture:
- Bend Estimation Module: Takes the binary piano rolls and predicts the continuous bend roll.
- Synthesis Module: Takes both the binary piano rolls and the predicted bend roll to generate a highly detailed mel spectrogram.
- Model Details: Both modules use a combination of Gated Recurrent Units (GRUs) and Transformer Encoders to extract note features. The denoiser is based on a non-causal WaveNet architecture. Feature-wise Linear Modulation (FiLM) layers are utilized to inject performer embeddings, allowing the model to capture unique stylistic and acoustic characteristics. Finally, a SoundStream vocoder converts the generated mel spectrogram into raw audio.
Datasets
The model was trained and evaluated on a recently published solo violin dataset (Tamer et al., 2023). After filtering for quality, the dataset contained 961 audio recordings (approximately 34 hours) across 21 distinct performers. It features etudes by Wohlfahrt and Kayser, and caprices by Paganini. Crucially, the dataset includes synchronized MIDI files with detailed multi-pitch bend information obtained via automatic transcription and alignment.
Key Insights and Ideas
- Polyphonic F0 as a Matrix: Instead of treating F0 as a 1D sequence, mapping pitch deviations to a 2D pitch-by-time "bend roll" elegantly solves the polyphonic representation problem.
- Explicit vs. Implicit Modeling: Relying on a diffusion model to implicitly learn pitch trajectories from discrete MIDI notes results in a loss of fine-grained control. Explicitly separating pitch-bend prediction into a dedicated front-end module dramatically improves the clarity and naturalness of expressive micro-tonal variations.
- Quantifying Vibrato: The authors introduce a novel evaluation metric for vibrato by applying a Discrete Fourier Transform (DFT) to the predicted F0 contours. This allows them to objectively measure both the presence of vibrato (F1 score) and stylistic accuracy (Perf-MAE) across different models.
Key Results
- Objective Metrics: ViolinDiff consistently outperformed the baseline model (which lacked explicit pitch bend conditioning) across all Fréchet Audio Distance (FAD) variants and vibrato metrics, proving that explicit conditioning helps capture performer-specific and piece-specific qualities.
- Visualizing Vibrato: Spectrogram analysis shows that ViolinDiff generates significantly clearer and more distinct vibrato waveforms in polyphonic sections compared to the baseline.
- Subjective Evaluation: In a MUSHRA realism listening test, ViolinDiff achieved a mean score of 70.96, significantly outperforming the baseline (66.99) as well as prior state-of-the-art diffusion models by Maman et al. (59.47) and Hawthorne et al. (43.67).
Limitations and Future Work
While ViolinDiff excels at pitch-based expressiveness, it currently lacks explicit controls for other vital performance parameters such as tempo dynamics and articulation (e.g., staccato vs. legato). The authors note that incorporating these control parameters is a key direction for future work. Additionally, they aim to extend this framework to synthesize other polyphonic instruments beyond the violin.
Summary updated