Learning to Solve Inverse Problems for Perceptual Sound Matching

Han Han, Vincent Lostanlen, Mathieu Lagrange

This blog-style summary breaks down the recent research on Perceptual Sound Matching (PSM), a task that sits at the intersection of signal processing, acoustics, and deep learning.

TL;DR

The authors introduce Perceptual-Neural-Physical (PNP) loss, a novel training objective that linearizes the relationship between synthesizer parameters and auditory features. This approach achieves a 100-fold speedup in training sound-matching models while enabling the use of high-fidelity perceptual features like the Joint Time-Frequency Scattering (JTFS) transform for state-of-the-art results.

What Problem Does This Paper Address?

Perceptual Sound Matching (PSM) is the task of finding the exact synthesizer settings (knobs, sliders, physical parameters) needed to recreate a specific target sound. It’s a classic inverse problem with applications in sound design, digital instrument modeling, and automatic orchestration.

The challenge lies in the "perceptual" part. Standard loss functions, like mean squared error on raw parameters, don't reflect how humans actually hear differences in timbre or rhythm. While Differentiable Digital Signal Processing (DDSP) allows training neural networks with perceptual losses (like spectrogram distance), it is computationally grueling. At every training step, the model must run the full synthesizer and a complex auditory feature transform. This makes using advanced, perceptually-accurate features like JTFS—which are computationally heavy—practically impossible for large-scale training.

Methods

The core innovation is the PNP loss, which sidesteps the heavy computation of DDSP by using a local approximation.

  1. Linearization via Taylor Expansion: The authors assume the mapping from synthesizer parameters to auditory features is locally smooth. They use a second-order Taylor expansion to approximate the perceptual loss as a quadratic form.
  2. Riemannian Metric Precomputation: Instead of running the synth and feature map during training, they precompute a Jacobian matrix for each training sample. This matrix is used to form a Riemannian metric $M(\theta)$ that captures how sensitive the sound is to changes in each parameter.
  3. Caching: These metric matrices are small and can be cached. During training, the loss calculation becomes a simple matrix-vector multiplication, removing the synthesizer and feature transform from the loop entirely.
  4. Joint Time-Frequency Scattering (JTFS): The authors use JTFS as their primary auditory descriptor. Unlike standard spectrograms, JTFS captures "mesostructures"—long-range temporal and frequency modulations that are crucial for identifying non-stationary sounds like chirps or drum hits.
  5. Adaptive Damping: To handle cases where the inverse problem is ill-conditioned (e.g., two parameters have nearly identical effects), they implement a Levenberg-Marquardt-style regularization that adaptively transitions from a simple parameter loss to the full perceptual PNP loss.
Graphical outline of the proposed PNP method compared to standard DDSP.

Datasets

The researchers evaluated their method on two challenging synthetic datasets of non-stationary sounds:

  • AM/FM Arpeggiator: An amplitude-modulated ascending chirp signal. This tests the model's ability to capture complex temporal envelopes and frequency sweeps.
  • FTM Drum Synthesizer: A physical model of a 2D rectangular membrane based on the Functional Transformation Method. This produces inharmonic, transient sounds with uneven decay across different partials, mimicking real-world percussive instruments.

Key Insights and Ideas

  • Local Geometry is Enough: You don't need to simulate the entire synthesis process at every training step. A local quadratic approximation of the perceptual space is sufficient to guide a neural network toward the right parameters.
  • Spectrograms Aren't Enough: The standard Multi-Scale Spectrogram (MSS) loss used in many DDSP models fails to capture the long-range periodicity and frequency evolution of complex sounds. JTFS is far superior for these "non-stationary" signals.
  • The "Mesostructure" Gap: MSS loss provides uninformative gradients for parameters like modulation frequency ($f_m$). JTFS fills this gap by analyzing temporal modulation rates directly.
  • Domain Knowledge Matters: Simple tricks like log-scaling parameters and min-max normalization significantly improve convergence, especially for physical models with wide parameter ranges.

Key Results

  • 100x Speedup: The PNP loss offers a massive computational advantage, speeding up the gradient descent process by two orders of magnitude compared to full DDSP.
  • State-of-the-Art Accuracy: The proposed model (combining P-loss pretraining with PNP-JTFS finetuning) achieved the lowest error rates on both datasets, significantly outperforming previous methods based on MSS or simple parameter loss.
  • Ablation Success: The ablation study (Figure 2) confirmed that the choice of JTFS as the feature map was the single most important factor in the model's performance.
Ablation study showing the impact of removing various components of the PNP pipeline. Lower values indicate better perceptual accuracy.

Limitations and Future Work

  • Supervised Requirement: PNP currently requires a synthetic dataset where the ground-truth parameters are known for training.
  • Differentiability: The synthesizer and feature map must be differentiable to allow for the initial Jacobian precomputation.
  • The Real-World Gap: While the model excels on synthetic targets, applying it to real-world recordings (where the exact synthesizer used is unknown) remains a challenge. The authors suggest domain adaptation as a key area for future research to bridge this gap.

Summary based on "Learning to Solve Inverse Problems for Perceptual Sound Matching" by Han et al. (2024).

Summary updated

Abstract

Perceptual sound matching (PSM) aims to find the input parameters to a synthesizer so as to best imitate an audio target. Deep learning for PSM optimizes a neural network to analyze and reconstruct prerecorded samples. In this context, our article addresses the problem of designing a suitable loss function when the training set is generated by a differentiable synthesizer. Our main contribution is perceptual-neural-physical loss (PNP), which aims at addressing a tradeoff between perceptual relevance and computational efficiency. The key idea behind PNP is to linearize the effect of synthesis parameters upon auditory features in the vicinity of each training sample. The linearization procedure is massively paralellizable, can be precomputed, and offers a 100-fold speedup during gradient descent compared to differentiable digital signal processing (DDSP). We demonstrate PNP on two datasets of nonstationary sounds: an AM/FM arpeggiator and a physical model of rectangular membranes. We show that PNP is able to accelerate DDSP with joint time-frequency scattering transform (JTFS) as auditory feature, while preserving its perceptual fidelity. Additionally, we evaluate the impact of other design choices in PSM: parameter rescaling, pretraining, auditory representation, and gradient clipping. We report state-of-the-art results on both datasets and find that PNP-accelerated JTFS has greater influence on PSM performance than any other design choice.
BibTeX
@inproceedings{han2024learning,
  author = {Han Han and Vincent Lostanlen and Mathieu Lagrange},
  title = {{Learning to Solve Inverse Problems for Perceptual Sound Matching}},
  booktitle = {IEEE/ACM},
  year = {2024},
  doi = {10.1109/taslp.2024.3393738},
}
Figure