Abstract
Perceptual sound matching (PSM) aims to find the input parameters to a synthesizer so as to best imitate an audio target. Deep learning for PSM optimizes a neural network to analyze and reconstruct prerecorded samples. In this context, our article addresses the problem of designing a suitable loss function when the training set is generated by a differentiable synthesizer. Our main contribution is perceptual-neural-physical loss (PNP), which aims at addressing a tradeoff between perceptual relevance and computational efficiency. The key idea behind PNP is to linearize the effect of synthesis parameters upon auditory features in the vicinity of each training sample. The linearization procedure is massively paralellizable, can be precomputed, and offers a 100-fold speedup during gradient descent compared to differentiable digital signal processing (DDSP). We demonstrate PNP on two datasets of nonstationary sounds: an AM/FM arpeggiator and a physical model of rectangular membranes. We show that PNP is able to accelerate DDSP with joint time-frequency scattering transform (JTFS) as auditory feature, while preserving its perceptual fidelity. Additionally, we evaluate the impact of other design choices in PSM: parameter rescaling, pretraining, auditory representation, and gradient clipping. We report state-of-the-art results on both datasets and find that PNP-accelerated JTFS has greater influence on PSM performance than any other design choice.
Introduction
In sound design, Perceptual Sound Matching (PSM) involves identifying the specific synthesizer parameters—such as oscillator frequency or filter cutoff—needed to recreate a target sound. This constitutes a classic inverse problem: while synthesis (mapping parameters to sound) is straightforward, the inverse mapping is computationally intensive.
Existing deep learning approaches typically face a trade-off. Parameter Loss (P-loss) is computationally efficient but fails to account for human auditory perception, where small parameter changes can lead to significant perceptual shifts. Conversely, Spectral or Perceptual Loss (DDSP) is highly accurate but suffers from extreme latency; it requires executing the entire synthesizer and auditory model during every backpropagation step. This research introduces the Perceptual-Neural-Physical (PNP) loss as a "middle way," combining perceptual awareness with the computational efficiency of parameter loss.
The Perceptual-Neural-Physical (PNP) Loss
The core innovation of this work is the PNP loss, which utilizes a second-order Taylor expansion to approximate perceptual loss. Instead of recalculating the complex relationship between parameters and sound at every training step, the method precomputes a Riemannian metric ($M$). This metric serves as a local map, weighting parameter errors based on their perceptual significance in specific regions of the parameter space.
Key technical components include:
- Precomputation: The metric $M$ is calculated once before training and cached, removing the synthesizer simulation from the training loop and enabling a 100-fold speedup.
- Joint Time-Frequency Scattering (JTFS): The model employs JTFS as its primary auditory feature, which excels at capturing the evolving textures of non-stationary sounds.
- Adaptive Damping: A damping term ($\lambda$), inspired by the Levenberg-Marquardt algorithm, ensures stability when multiple parameter combinations produce similar sounds.
Experimental Evaluation
The PNP loss was evaluated on two challenging datasets of non-stationary sounds:
- AM/FM Arpeggiator: Tests the model's ability to track rapid frequency modulations and rhythmic patterns.
- FTM Drum Synthesizer: A physical model of a rectangular membrane producing complex, inharmonic transient sounds with non-linear frequency decay.
Results and Discussion
The PNP-accelerated model using JTFS consistently achieved state-of-the-art performance on both datasets. Beyond the 100-fold training speedup, ablation studies revealed that the choice of auditory representation is the most critical factor in PSM performance.
As shown in the ablation results, replacing JTFS with a standard Multi-Scale Spectrogram (MSS) leads to a significant degradation in performance. Other factors, such as parameter rescaling (min-max scaling) and gradient clipping, also contribute to the final model's accuracy, but the combination of PNP and JTFS remains the most influential design choice.
Conclusion
The PNP loss demonstrates that local linearization of the parameter-to-sound relationship is a powerful tool for efficient perceptual sound matching. While currently limited to supervised learning on synthetic datasets with known ground-truth parameters, future work will explore domain adaptation to apply these techniques to real-world recordings. This approach paves the way for real-time, perceptually-accurate sound analysis tools for musicians and sound designers.
Summary updated