SerumRNN: Step by Step Audio VST Effect Programming

Christopher Mitcheltree, Hideki Koike

TL;DR

SerumRNN is an AI-powered assistant designed to help sound designers navigate the complex world of VST (Virtual Studio Technology) synthesizers. Unlike "black-box" tools that generate a sound in one go, SerumRNN provides a transparent, step-by-step roadmap of audio effects and parameter settings to transform a starting sound into a desired target.

What Problem Does This Paper Address?

Sound design is notoriously difficult. Professional-grade synthesizers like Xfer Records' Serum offer nearly infinite combinations of oscillators, filters, and effects. For beginners, achieving a specific "timbre" (the character or quality of a sound) usually involves hours of frustrating trial and error.

While previous research has used AI to automate this process, most existing systems are "one-shot" models. They take a target sound and spit out a final configuration. This approach has two major flaws:

  1. Lack of Interpretability: The user doesn't learn how the sound was made.
  2. Lack of Control: The user cannot easily intervene or tweak the process at intermediate stages.

The authors address this by creating a "white-box" system that mimics a human expert, suggesting one effect at a time and explaining exactly how to tune it.

Methods

The researchers built an ensemble of neural networks to handle the two distinct parts of the problem:

  1. Effect Selection Model: A Recurrent Neural Network (specifically a bi-directional LSTM) that looks at the current sound and the target sound to decide which effect (e.g., Distortion, Reverb, EQ) should be applied next. It also considers which effects have already been used to avoid redundancy.
  2. Effect Parameter Models: A collection of five specialized Convolutional Neural Networks (CNNs)—one for each supported effect. Once an effect is selected, the corresponding CNN determines the specific "knob settings" (parameters) required to move the current sound closer to the target.

The system uses two types of audio features as input: Mel spectrograms (visual representations of frequency over time) and MFCCs (Mel-frequency cepstral coefficients), which help the model understand the "shape" of the sound's spectrum.

Datasets

Because Serum is a software synthesizer, the authors were able to generate a massive, high-quality dataset:

  • The Source: 1.536 million audio clips rendered directly from Serum.
  • Complexity Tiers: The data was split into three groups of "presets": Basic Shapes (simple waves), Advanced Shapes (dual oscillators), and Advanced Modulating Shapes (complex, moving sounds).
  • Effects: The model was trained on five common effects: multi-band compression, distortion, EQ, phaser, and hall reverb.
  • Standardization: All samples were rendered at a C4 pitch for one second to focus purely on timbre rather than melody.

Key Insights and Ideas

  • Iterative Learning is Better than One-Shot: By modifying the audio step-by-step, the model receives "feedback" at every stage. This makes the system more robust than models that try to guess all parameters simultaneously.
  • The "Oracle" Paradox: In a surprising twist, SerumRNN often outperformed the "Oracle" (the actual sequence of effects used to create the target sound). This suggests the AI discovered more efficient ways to reach a sonic goal than the original human-programmed preset.
  • Prioritization: The model naturally learned to apply the most "dramatic" effects first (like distortion or heavy compression) and leave subtle refinements (like reverb) for the final steps, mirroring how professional sound designers work.
  • White-Box Education: Because the system provides intermediate steps, it functions as an educational tool, showing the user exactly how each effect contributes to the final texture.

Key Results

The authors evaluated the system using several metrics, most notably MSSMAE (Multi-scale spectral mean absolute error), which measures how closely the frequency content matches the target.

  • Accuracy: The Effect Selection Model achieved over 98% accuracy in predicting the next logical step in a sequence.
  • Error Reduction: Across all complexity tiers, SerumRNN significantly reduced the distance between the starting audio and the target. In the "Advanced Modulating" category, it achieved an 81% reduction in Mean Squared Error (MSE).
  • Speed: The system is highly efficient, with an inference time of approximately 300ms per effect, making it fast enough for near-real-time use in a creative workflow.

Limitations and Future Work

  • Musicality: The training data was sampled randomly. While this covers a wide mathematical space, many of the resulting sounds are "unmusical" or harsh. Future versions could use adversarial training to focus on sounds that humans actually find pleasing.
  • Phase Information: The model currently struggles with the Phaser effect because spectrograms don't represent "phase" (the timing of wave peaks) very well.
  • Integration: The authors plan to package SerumRNN as a Max for Live plugin, allowing music producers to use it directly inside Ableton Live.
  • User Studies: While the quantitative metrics are strong, the authors acknowledge the need for qualitative studies to see if beginners actually find the tool helpful for learning.

Summary updated

Abstract

Learning to program an audio production VST synthesizer is a time consuming process, usually obtained through inefficient trial and error and only mastered after years of experience. As an educational and creative tool for sound designers, we propose SerumRNN: a system that provides step-by-step instructions for applying audio effects to change a user’s input audio towards a desired sound. We apply our system to Xfer Records Serum: currently one of the most popular and complex VST synthesizers used by the audio production community. Our results indicate that SerumRNN is consistently able to provide useful feedback for a variety of different audio effects and synthesizer presets. We demonstrate the benefits of using an iterative system and show that SerumRNN learns to prioritize effects and can discover more efficient effect order sequences than a variety of baselines.
BibTeX
@inproceedings{mitcheltree2021serumrnn,
  author = {Christopher Mitcheltree and Hideki Koike},
  title = {{SerumRNN: Step by Step Audio VST Effect Programming}},
  booktitle = {EvoMUSART},
  year = {2021},
  url = {https://openreview.net/forum?id=DRfMP-Mrbq},
}

Loading related content...

Figure