SerumRNN: Step by Step Audio VST Effect Programming

Christopher Mitcheltree, Hideki Koike

Abstract

Learning to program an audio production VST synthesizer is a time consuming process, usually obtained through inefficient trial and error and only mastered after years of experience. As an educational and creative tool for sound designers, we propose SerumRNN: a system that provides step-by-step instructions for applying audio effects to change a user's input audio towards a desired sound. We apply our system to Xfer Records Serum: currently one of the most popular and complex VST synthesizers used by the audio production community. Our results indicate that SerumRNN is consistently able to provide useful feedback for a variety of different audio effects and synthesizer presets. We demonstrate the benefits of using an iterative system and show that SerumRNN learns to prioritize effects and can discover more efficient effect order sequences than a variety of baselines.

Introduction

Sound design is notoriously difficult. Professional-grade synthesizers like Xfer Records' Serum offer nearly infinite combinations of oscillators, filters, and effects. For beginners, achieving a specific "timbre" (the character or quality of a sound) usually involves hours of frustrating trial and error.

While previous research has used AI to automate this process, most existing systems are "one-shot" models that generate a final configuration in a single step. This approach suffers from a lack of interpretability—users don't learn how the sound was constructed—and a lack of control, as users cannot easily intervene at intermediate stages. SerumRNN addresses this by creating a "white-box" system that mimics a human expert, suggesting one effect at a time and explaining exactly how to tune it.

System Architecture

The researchers built an ensemble of neural networks to handle the two distinct parts of the sound design problem: effect selection and parameter optimization.

  1. Effect Selection Model: A Recurrent Neural Network (specifically a bi-directional LSTM) analyzes the current sound and the target sound to decide which effect (e.g., Distortion, Reverb, EQ) should be applied next. It maintains a history of used effects to avoid redundancy.
  2. Effect Parameter Models: A collection of five specialized Convolutional Neural Networks (CNNs)—one for each supported effect—determines the specific parameter settings required to move the current sound closer to the target once an effect is selected.
The SerumRNN system architecture utilizes dual-input feature extraction (MFCC and Mel Spectrogram) to drive an iterative effect selection and parameterization loop.

The system utilizes two types of audio features as input: Mel spectrograms (visual representations of frequency over time) and MFCCs (Mel-frequency cepstral coefficients), which help the model understand the spectral "shape" of the audio.

Dataset and Training

Because Serum is a software synthesizer, the authors generated a massive, high-quality dataset of 1.536 million audio clips. The data was categorized into three complexity tiers: Basic Shapes (simple waves), Advanced Shapes (dual oscillators), and Advanced Modulating Shapes (complex, moving sounds). All samples were rendered at a C4 pitch for one second to focus purely on timbre. The model was trained on five common effects: multi-band compression, distortion, EQ, phaser, and hall reverb.

Key Insights and Results

SerumRNN demonstrates that iterative learning is superior to one-shot estimation. By modifying the audio step-by-step, the model receives feedback at every stage, making the system more robust.

An example of the iterative process where the system sequentially applies Phaser, Distortion, and Equalizer to transform the original audio into a target spectrogram.

Key findings include:

  • Efficiency: SerumRNN often outperformed the "Oracle" (the actual sequence used to create the target), discovering more efficient ways to reach a sonic goal than the original human-programmed preset.
  • Prioritization: The model naturally learned to apply dramatic effects (distortion, compression) first, leaving subtle refinements (reverb) for the final steps, mirroring professional workflows.
  • Performance: The Effect Selection Model achieved over 98% accuracy. In the most complex "Advanced Modulating" category, the system achieved an 81% reduction in Mean Squared Error (MSE) relative to the target sound.
  • Speed: With an inference time of approximately 300ms per effect, the system is fast enough for near-real-time creative use.

Conclusion

SerumRNN provides a transparent, educational roadmap for sound designers. While the current model focuses on a subset of effects and can occasionally struggle with phase-heavy effects like Phasers, it represents a significant step toward interpretable AI assistants in music production. Future work involves expanding the effect library and integrating the system directly into digital audio workstations (DAWs) via plugins like Max for Live.

Summary updated

Abstract

Learning to program an audio production VST synthesizer is a time consuming process, usually obtained through inefficient trial and error and only mastered after years of experience. As an educational and creative tool for sound designers, we propose SerumRNN: a system that provides step-by-step instructions for applying audio effects to change a user's input audio towards a desired sound. We apply our system to Xfer Records Serum: currently one of the most popular and complex VST synthesizers used by the audio production community. Our results indicate that SerumRNN is consistently able to provide useful feedback for a variety of different audio effects and synthesizer presets. We demonstrate the benefits of using an iterative system and show that SerumRNN learns to prioritize effects and can discover more efficient effect order sequences than a variety of baselines.
BibTeX
@inproceedings{mitcheltree2021serumrnn,
  author = {Christopher Mitcheltree and Hideki Koike},
  title = {{SerumRNN: Step by Step Audio VST Effect Programming}},
  booktitle = {EvoMUSART},
  year = {2021},
  url = {https://openreview.net/forum?id=DRfMP-Mrbq},
}
Figure