Differentiable Artificial Reverberation

Sungho Lee, Hyeong-Seok Choi, Kyogu Lee

TL;DR

This paper introduces Differentiable Artificial Reverberation (DAR) models, bridging the gap between classical digital signal processing (DSP) and modern deep learning. By replacing slow, recursive Infinite Impulse Response (IIR) filters with GPU-friendly Finite Impulse Response (FIR) approximations via frequency sampling, the authors enable end-to-end training for reverberation parameter estimation, significantly outperforming traditional non-differentiable baselines.

What Problem Does This Paper Address?

Artificial Reverberation (AR) models are essential for creating realistic audio environments, but tuning their parameters (ARPs) to match a specific room is difficult. Traditionally, this was done via "parameter matching," where a neural network tries to predict the exact numerical settings of a reverb plugin.

However, this approach has two major flaws:

  1. Lack of End-to-End gradients: You can't backpropagate through a standard reverb plugin. This means the network can't "hear" the result of its predictions during training; it only knows if the numbers match, not if the sound matches.
  2. The GPU Bottleneck: AR models often rely on IIR filters, which are recursive (each output depends on the previous one). This makes them incredibly slow to run on GPUs, which prefer parallel operations.

Methods

The authors propose a "differentiable" version of three popular AR models: Filtered Velvet Noise (FVN), Advanced FVN (AFVN), and Delay Networks (DN).

The key technical innovation is the Frequency-Sampling Method. Instead of implementing the recursive difference equations directly, they sample the filter's frequency response and convert it into a high-order FIR filter. This allows the reverberation to be generated using Fast Fourier Transforms (FFTs), which are highly optimized for GPUs.

The Architecture: * Encoder: A combination of 2D Convolutional layers and Gated Recurrent Units (GRUs) that takes either a Room Impulse Response (RIR) or reverberant speech and produces a latent representation. * DAR Layer: The predicted parameters are fed into the differentiable reverb model to generate a synthetic RIR. * Loss Function: A multi-scale spectral loss compares the generated RIR to the target. Because the DAR layer is differentiable, the error can flow all the way back to the encoder.

Datasets

  • Synthetic Data: 200,000 RIRs generated using the "shoebox" image-source method for training.
  • Real-World Data: 1,835 real-world RIR measurements from the OpenAIR and ACE Challenge datasets (used for validation and testing).
  • Speech: The VCTK dataset was used to provide "dry" speech for the blind estimation tasks.

Key Insights and Ideas

  • The "DSP Prior" Advantage: While a pure CNN-based "black box" decoder can achieve lower mathematical loss, it often produces audible artifacts. The DAR models act as a "DSP Prior"—they constrain the output to sound like a real room, leading to much higher subjective quality scores.
  • Frequency Sampling as a Bridge: The paper proves that frequency sampling introduces negligible error (mostly in low-energy regions) while providing the massive speedup needed for deep learning.
  • Task Agnosticism: Because the model learns to match the sound (the RIR) rather than the parameters, the same architecture can handle both "analysis-synthesis" (matching a known RIR) and "blind estimation" (extracting reverb from a speech recording).

Key Results

  • Objective Metrics: The DAR models outperformed parameter-matching baselines by a wide margin. For example, the error in reverberation time ($\Delta T_{30}$) was reduced from over 20% in baselines to under 5% with DAR.
  • Subjective Quality: In MUSHRA listening tests, the proposed end-to-end method scored between 70-80 (on a 100-point scale), indicating a "noticeable but not much" difference from the reference, whereas non-differentiable baselines scored below 60.
  • Efficiency: The models are efficient enough for real-time use on a CPU after training, with the AFVN model requiring only ~1511 FLOPs per sample.

Limitations and Future Work

  • Simulation-to-Real Gap: The authors noted that performance drops slightly when moving from synthetic "shoebox" rooms to complex real-world environments. They suggest that more diverse training data or better augmentation is needed.
  • Expressivity Trade-offs: There is a clear trade-off between model complexity and ease of training. The FVN model is the most flexible but has the most parameters (930), while the Delay Network is more compact (200 parameters) but harder to optimize due to its restricted "forced exponential decay" structure.
  • Future Directions: The authors suggest exploring "AR-model-friendly" neural network architectures that could yield even better performance-efficiency trade-offs.

Summary updated

Abstract

Artificial reverberation (AR) models play a central role in various audio applications. Therefore, estimating the AR model parameters (ARPs) of a reference reverberation is a crucial task. Although a few recent deep-learning-based approaches have shown promising performance, their non-end-to-end training scheme prevents them from fully exploiting the potential of deep neural networks. This motivates the introduction of differentiable artificial reverberation (DAR) models, allowing loss gradients to be back-propagated end-to-end. However, implementing the AR models with their difference equations “as is” in the deep learning framework severely bottlenecks the training speed when executed with a parallel processor like GPU due to their infinite impulse response (IIR) components. We tackle this problem by replacing the IIR filters with finite impulse response (FIR) approximations with the frequency-sampling method. Using this technique, we implement three DAR models—differentiable Filtered Velvet Noise (FVN), Advanced Filtered Velvet Noise (AFVN), and Delay Network (DN). For each AR model, we train its ARP estimation networks for analysis-synthesis (RIR-to-ARP) and blind estimation (reverberant-speech-to-ARP) task in an end-to-end manner with its DAR model counterpart. Experiment results show that the proposed method achieves consistent performance improvement over the non-end-to-end approaches in both objective metrics and subjective listening test results.
BibTeX
@article{lee2022differentiable,
  author = {Sungho Lee and Hyeong-Seok Choi and Kyogu Lee},
  title = {{Differentiable Artificial Reverberation}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=gMqlnR99s7},
}

Loading related content...

Figure