Abstract
Generalization of synthesizer sound matching to external instrument sounds is highly challenging due to the non-differentiability of sound synthesis process which prohibits the use of out-of-domain sounds for training with synthesis parameter loss. We propose SynthRL, a novel reinforcement learning (RL)-based approach for cross-domain synthesizer sound matching. By incorporating sound similarity into the reward function, SynthRL effectively optimizes synthesis parameters without ground-truth labels, allowing fine-tuning on out-of-domain sounds. Furthermore, we introduce a transformer-based model architecture and reward-based prioritized experience replay to enhance RL training efficiency, considering the unique characteristics of the task. Experimental results demonstrate that SynthRL outperforms state-of-the-art methods on both in-domain and out-of-domain tasks. Further experimental analysis validates the effectiveness of our reward design, showing a strong correlation with human perception of sound similarity.
Introduction
In music production, "sound design"—the process of adjusting a synthesizer's parameters to create a specific texture—is a skill that takes years to master. Automating this via "synthesizer sound matching" is a significant AI challenge because the sound generation process is non-differentiable. Traditional models rely on "parameter loss," learning from ground-truth knob settings of existing presets. While effective for "in-domain" sounds (those generated by the specific synth used for training), these models fail to generalize to "out-of-domain" sounds, such as real instruments or different synthesizers, where ground-truth parameters do not exist.
Model Architecture
SynthRL addresses this by treating sound matching as a reinforcement learning problem. The system utilizes a Transformer-based encoder-decoder architecture. A 2D CNN extracts features from the target sound's mel-spectrogram, which are then processed by a non-autoregressive Transformer decoder to predict synthesizer parameters (oscillators, filters, envelopes) as a classification task.
Reinforcement Learning Framework
Because the synthesizer acts as a non-differentiable "black box," SynthRL uses RL to optimize parameters based on a perceptual reward. The model acts as a policy that predicts parameter settings; these settings generate audio, which is then compared to the target sound to calculate a reward.
To mimic human hearing, the reward function combines three spectral metrics: Spectrogram Mean Absolute Error (MAE), Spectral Convergence, and MFCC MAE. This multi-metric approach prevents the model from converging on "lazy" solutions, such as short or quiet sounds that might minimize a single error metric but fail to capture timbre. Additionally, the authors adapted Prioritized Experience Replay (PER) to selectively replay high-reward actions, helping the model navigate the complex, "spiky" parameter spaces of modern synthesizers.
Training Strategy
The framework employs a three-stage training pipeline to bridge the gap between supervised learning and cross-domain generalization:
- Stage 1: Supervised learning on in-domain data using parameter loss.
- Stage 2: A hybrid stage combining parameter loss and RL reward.
- Stage 3: Pure RL fine-tuning on out-of-domain sounds, where no ground-truth labels are available.
Experimental Evaluation
The model was tested using the Dexed synthesizer (FM synthesis) for in-domain data and the Surge XT synthesizer (subtractive hybrid synthesis) for out-of-domain data. SynthRL significantly outperformed state-of-the-art baselines like Sound2Synth and PresetGenVAE.
In quantitative tests on out-of-domain tasks, SynthRL achieved the lowest error rates across all spectral metrics. Qualitative human listening tests (Mean Opinion Score) further validated these results; for out-of-domain sounds, SynthRL scored 3.01, while the leading baseline scored only 2.09. In head-to-head ABX tests, users preferred SynthRL’s matches over 70% of the time for cross-domain challenges.
Conclusion
SynthRL demonstrates that reinforcement learning can effectively bridge the gap in synthesizer sound matching where differentiability is absent. By leveraging perceptual rewards and a robust Transformer architecture, the model generalizes to diverse sound sources. Future work will look to expand this framework to handle polyphonic sounds, varying pitches, and more complex temporal dynamics beyond single-note matching.
Summary updated