TL;DR
This paper introduces a system capable of "reverse-engineering" the complex chain of effects applied to an audio track—known as an audio processing graph—using only the final processed audio as a reference. By combining a convolutional encoder with a transformer-based graph decoder, the model can reconstruct both the arrangement of processors (like filters, distortion, and reverb) and their specific numerical parameters.
What Problem Does This Paper Address?
In professional audio production, sound is rarely "dry." It passes through a complex web of processors—equalizers, compressors, delays, and more—connected in specific routings. While previous AI research has focused on estimating the settings of a fixed set of effects, this paper tackles the much harder "blind estimation" problem: identifying the structure of the graph itself.
The challenge is significant because the search space is effectively infinite, many different graph configurations can produce similar sounds (the one-to-many problem), and some effects (like heavy distortion) can "mask" the presence of processors that came before them in the chain.
Methods
The authors propose a two-stage deep learning framework:
- Reference Encoder: A VGGish-style convolutional neural network (CNN) processes the Mel spectrogram of the "wet" (processed) audio. It uses attentive pooling to compress the audio features into a latent vector $z$.
- Prototype Decoder (TokenGT): To handle the graph structure, the authors treat the graph as a sequence of tokens. Using a Transformer-based architecture, the model predicts nodes (processors) and edges (connections) one by one in a specific "breadth-first search" (BFS) order, starting from the output node and working backward.
- Parameter Estimator: Once the "prototype" graph (the structure) is decided, a second pass uses a similar architecture to estimate the continuous-valued parameters (e.g., the exact frequency of a filter or the threshold of a compressor).
To handle the ambiguity where different orders of linear effects (like two EQs in a row) produce the same result, they implement LTI Reordering, a rule-based system to standardize the ground-truth graphs during training.
Datasets
Since no public dataset of "audio processing graphs" exists, the authors created their own:
- Synthetic Graph Generator: They used 33 conventional digital signal processing (DSP) modules to build random but "realistic" graphs based on common production "motifs" (e.g., gated reverb, multiband distortion).
- Singing Dataset: 300,000 pairs based on the OpenSinger dataset.
- Drum Dataset: 450,000 pairs using commercial drum samples and MIDI files, including complex "mixing bus" routings.
Key Insights and Ideas
- Graph as a Language: By treating graph components (nodes and edges) as tokens in a sequence, the authors successfully applied standard Transformer-based language modeling techniques to a complex structural problem.
- The "Masking" Effect: The authors discovered that non-linear processors like
bitcrushact as information bottlenecks. If a bitcrusher is set to a very low bit depth, it destroys the signal so thoroughly that the model cannot "see" the filters that were applied before it. - Structure vs. Parameters: Separating the task into "what is the graph?" and "what are the settings?" proved more effective than trying to estimate everything in a single stage.
- Sparsity Matters: The authors noted that audio graphs are much sparser than chemical or social graphs. This means standard graph neural networks (GNNs) that rely on local neighborhood aggregation struggle to capture the global flow of the signal, justifying their choice of a Transformer.
Key Results
- Perceptual Success: While the model rarely reconstructed the exact original graph (perfect reconstruction is rare), the audio rendered from the estimated graphs was perceptually very close to the original. In MUSHRA (subjective listening) tests, the system scored 76.6/100 for singing and 69.8/100 for drums.
- Structural Accuracy: On "seen" speaker/kit sets, the model achieved a node error rate of 0.215 for singing and 0.335 for drums.
- Ablation: The "token-by-token" decoding strategy significantly outperformed traditional "node-by-node" graph generation methods, particularly in maintaining a low "invalid graph rate" (ensuring the predicted graph actually functions).
Limitations and Future Work
- Toy Examples: The current processors are "default" DSP implementations. Real-world plugins (like a vintage 1176 compressor) have unique non-linear characteristics not yet captured here.
- Generalization: The model's performance drops when encountering "unseen" dry audio sources, suggesting the encoder is still somewhat sensitive to the underlying content rather than just the effects.
- Differentiable DSP: The authors suggest that moving toward an end-to-end system using differentiable processors could allow the model to fine-tune its estimates based on the final audio output, potentially solving the "masking" issues.
Summary updated