Blind Estimation of Audio Processing Graph

Sungho Lee, Jaehyun Park, Seungryeol Paik, Kyogu Lee

Abstract

Musicians and audio engineers sculpt and transform their sounds by connecting multiple processors, forming an audio processing graph. However, most deep-learning methods overlook this real-world practice and assume fixed graph settings. To bridge this gap, we develop a system that reconstructs the entire graph from a given reference audio. We first generate a realistic graph-reference pair dataset and train a simple blind estimation system composed of a convolutional reference encoder and a transformer-based graph decoder. We apply our model to singing voice effects and drum mixing estimation tasks. Evaluation results show that our method can reconstruct complex signal routings, including multi-band processing and sidechaining.

Introduction

In professional audio production, the final sound is rarely the result of a single processor. Instead, it emerges from a complex web of interconnected effects—equalizers, compressors, delays, and reverbs—known as an audio processing graph. While previous research in music information retrieval (MIR) has focused on estimating the parameters of a fixed set of effects, this paper addresses the significantly more difficult "blind estimation" problem: identifying the underlying structure of the graph itself using only the processed audio as a reference.

This task presents several unique challenges. The search space for potential graph configurations is effectively infinite. Furthermore, the problem is one-to-many; different graph arrangements can often produce perceptually similar results. Finally, non-linear processors, such as heavy distortion or bitcrushing, can "mask" the characteristics of preceding effects in the chain, making it difficult for a model to recover the full signal path.

System Architecture

The proposed framework approaches graph reconstruction through a two-stage deep learning process: structure prediction followed by parameter estimation.

1. Reference Encoder

The system begins with a VGGish-style convolutional neural network (CNN) that processes the Mel spectrogram of the "wet" (processed) audio. Using attentive pooling, the encoder compresses these high-dimensional audio features into a compact latent vector $z$, which serves as the representation of the audio's sonic character.

2. Prototype Decoder (TokenGT)

To reconstruct the graph structure, the authors treat the graph as a sequence of tokens. Utilizing a Transformer-based architecture, the model predicts nodes (processors) and edges (connections) sequentially. To ensure a consistent representation, the graph is decoded using a breadth-first search (BFS) order, starting from the output node and working backward through the signal chain.

3. Parameter Estimator

Once the "prototype" graph structure is established, a second pass uses a similar Transformer architecture to estimate the continuous-valued parameters for each node (e.g., the cutoff frequency of a filter or the ratio of a compressor). To handle the ambiguity where different orders of linear time-invariant (LTI) effects produce identical results, the researchers implemented an "LTI Reordering" system to standardize ground-truth graphs during training.

Data Synthesis and Training

Because no large-scale public dataset of audio processing graphs exists, the authors developed a synthetic graph generator. This tool utilizes 33 conventional digital signal processing (DSP) modules to build random but musically realistic graphs based on common production "motifs," such as gated reverb or multiband distortion.

Two primary datasets were generated:

  • Singing Dataset: 300,000 pairs based on the OpenSinger dataset.
  • Drum Dataset: 450,000 pairs using commercial drum samples and MIDI files, featuring complex mixing bus routings and sidechaining.

Key Insights

Graph as a Language

By treating graph components as tokens in a sequence, the model successfully leverages standard Transformer-based language modeling techniques. This approach proved more effective than traditional Graph Neural Networks (GNNs). Because audio graphs are relatively sparse compared to chemical or social graphs, the global attention mechanism of a Transformer is better suited to capturing the signal flow than the local neighborhood aggregation used in GNNs.

The Information Bottleneck

The research highlighted a "masking" effect caused by non-linear processors. For example, if a bitcrusher is set to a very low bit depth, it destroys signal information so thoroughly that the model cannot accurately identify filters applied earlier in the chain. This suggests that certain graph configurations are inherently lossy from a reconstruction perspective.

Evaluation and Results

The model's performance was evaluated through both objective structural metrics and subjective listening tests (MUSHRA).

  • Perceptual Similarity: While the model rarely reconstructed the exact original graph, the audio rendered from the predicted graphs was perceptually very close to the reference. In MUSHRA tests, the system achieved scores of 76.6/100 for singing and 69.8/100 for drums.
  • Structural Accuracy: On seen speaker and drum kit sets, the model achieved a node error rate of 0.215 for singing and 0.335 for drums.
  • Decoding Strategy: The token-by-token BFS decoding strategy significantly outperformed node-by-node generation methods, particularly in maintaining a low "invalid graph rate," ensuring that predicted structures were technically functional.

Limitations and Future Directions

While the system demonstrates a breakthrough in blind graph estimation, several limitations remain. The current DSP modules are "default" implementations; they do not yet capture the unique non-linearities of specific vintage hardware or high-end plugins. Additionally, the model shows some sensitivity to the underlying "dry" audio content, suggesting that the encoder could be further refined to isolate effect characteristics from source material.

Future work may involve moving toward an end-to-end system using Differentiable DSP. This would allow the model to fine-tune its parameter estimates by directly minimizing the difference between the rendered output and the reference audio, potentially overcoming the masking issues identified in this study.

Summary updated

Abstract

Musicians and audio engineers sculpt and transform their sounds by connecting multiple processors, forming an audio processing graph. However, most deep-learning methods overlook this real-world practice and assume fixed graph settings. To bridge this gap, we develop a system that reconstructs the entire graph from a given reference audio. We first generate a realistic graph-reference pair dataset and train a simple blind estimation system composed of a convolutional reference encoder and a transformer-based graph decoder. We apply our model to singing voice effects and drum mixing estimation tasks. Evaluation results show that our method can reconstruct complex signal routings, including multi-band processing and sidechaining.
BibTeX
@inproceedings{lee2023blind,
  author = {Sungho Lee and Jaehyun Park and Seungryeol Paik and Kyogu Lee},
  title = {{Blind Estimation of Audio Processing Graph}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=o7wrxFrWvN},
}
Figure