DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge

Lenny Renault, Rémi Mignot, Axel RöebelJournal of the Audio Engineering Society 2023 · DOI

TL;DR

This paper introduces DDSP-Piano, a novel neural network for synthesizing realistic piano sounds that leverages instrument-specific knowledge within a differentiable framework. It achieves better sound quality than previous neural methods with fewer parameters, offering a more interpretable and controllable approach to digital piano synthesis.

What Problem Does This Paper Address?

Synthesizing realistic and controllable instrument sounds is a long-standing challenge in digital audio. While deep learning has made significant strides, many neural models act as black boxes, offering limited interpretability and control. Existing methods for piano synthesis often struggle with polyphony (multiple notes played simultaneously) or lack the nuanced control that musicians expect. The authors aim to bridge this gap by developing a polyphonic piano synthesizer that incorporates prior knowledge about piano acoustics into a differentiable neural network, allowing for better control and understanding of the synthesis process.

Methods

The core of DDSP-Piano is the Differentiable Digital Signal Processing (DDSP) framework, which allows traditional signal processing components (like synthesizers and filters) to be trained end-to-end with neural networks. DDSP-Piano extends this by:

  • Polyphonic Conditioning: It handles multiple simultaneous notes by splitting the input into individual "voices" and processing them in parallel.
  • Instrument Knowledge Integration: The model explicitly incorporates knowledge about piano acoustics, such as:
  • Inharmonicity: The fact that piano string partials are not perfect harmonics of the fundamental frequency.
  • Note Release: Modeling the gradual decay of sound after a key is released.
  • Detuning: Accounting for slight detuning between duplicated strings that causes beating.
  • Harmonic-Plus-Noise Synthesizer: The model uses a harmonic synthesizer for the main tonal components and a filtered noise synthesizer for percussive and transient sounds.
  • Differentiable Reverberation: A learned room impulse response is applied to simulate the recording environment.
  • Two-Phase Training: A novel training strategy is employed to optimize the neural network components and the explicit acoustic models separately before fine-tuning.

Datasets

The paper uses the MAESTRO dataset (v3.0.0) for training and evaluation. This dataset contains approximately 200 hours of professional piano performances, including MIDI data (notes, velocities, pedal information) and corresponding audio recordings. The audio is downsampled to 16kHz and downmixed to mono.

Key Insights and Ideas

  • Explicit knowledge beats black boxes: Incorporating explicit acoustic models (like inharmonicity) into a neural framework leads to better performance and interpretability than purely data-driven approaches.
  • DDSP is a powerful framework for instrument modeling: The differentiable nature of DDSP allows for seamless integration of signal processing and neural networks, enabling more controlled and interpretable synthesis.
  • "Double decay" is learned implicitly: The model naturally learns the characteristic two-stage decay of piano notes without explicit programming, demonstrating the power of recurrent layers in capturing temporal nuances.
  • Reverb can be a tricky component: The reverberation module, while differentiable, sometimes over-learned characteristics that should have been handled by other parts of the model, highlighting the challenge of disentangling sound components.
  • Two-phase training is crucial: Separating the optimization of frequency-related components from spectral envelope modeling improves training stability and final performance.

Key Results

  • Subjective Evaluation (MOS): DDSP-Piano achieved better sound quality ratings than the "Piano-TTS" neural benchmark, despite using significantly fewer parameters.
  • Objective Evaluation (Spectral Loss): The model demonstrated good reconstruction quality, with the explicit inharmonicity model proving valuable.
  • Comparison to Baselines: DDSP-Piano outperformed other neural methods and was comparable to physical-modeling-based systems (Pianoteq), though real recordings were still perceived as slightly better.
  • Computational Load: The current implementation is not real-time on CPU but shows promise for real-time performance with GPU acceleration or optimized inference frameworks.

Limitations and Future Work

  • Reverberation Over-expression: The reverberation module sometimes learned characteristics beyond the room acoustics, suggesting a need for better disentanglement and potentially more constrained reverb models.
  • Disentanglement Challenges: While the model architecture allows for it, achieving perfect disentanglement of sound components (e.g., separating harmonic and noise contributions) remains an area for improvement.
  • Extreme Instrument Properties: The model might struggle with instruments exhibiting extreme acoustic properties not present in the training data.
  • Real-time Inference: Further optimization is needed for real-time CPU inference.
  • Future Directions: The authors suggest leveraging the interpretability and differentiability of DDSP-Piano for tasks like audio source separation and self-supervised multi-pitch transcription. They also propose exploring physics-informed models that balance neural expressivity with explicit constraints.

Summary updated

Abstract

International audience
BibTeX
@article{renault2023ddsp,
  author = {Lenny Renault and Rémi Mignot and Axel Röebel},
  title = {{DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge}},
  journal = {Journal of the Audio Engineering Society},
  year = {2023},
  doi = {10.17743/jaes.2022.0102},
}
Figure