Abstract
General-purpose audio representations have proven effective across diverse music information retrieval applications, yet their utility in intelligent music production remains limited by insufficient understanding of audio effects (Fx). Although previous approaches have emphasized audio effects analysis at the mixture level, this focus falls short for tasks demanding instrument-wise audio effects understanding, such as automatic mixing. In this work, we present Fx-Encoder++, a novel model designed to extract instrument-wise audio effects representations from music mixtures. Our approach leverages a contrastive learning framework and introduces an "extractor" mechanism that, when provided with instrument queries (audio or text), transforms mixture-level audio effects embeddings into instrument-wise audio effects embeddings. We evaluated our model across retrieval and audio effects parameter matching tasks, testing its performance across a diverse range of instruments. The results demonstrate that Fx-Encoder++ outperforms previous approaches at mixture level and show a novel ability to extract effects representation instrument-wise, addressing a critical capability gap in intelligent music production systems.
Introduction
In the field of Intelligent Music Production, understanding how a sound was processed—identifying effects like reverb, distortion, or EQ—is as critical as identifying the instrument itself. While general-purpose models excel at instrument recognition, they often overlook the subtle timbral changes introduced by audio effects (Fx). Existing tools typically analyze either isolated tracks or entire mixtures, creating a significant gap for real-world applications like automatic mixing, where a producer needs to understand the processing of specific instruments within a dense mix.
Previously, analyzing instrument-specific effects required source separation to "unmix" the song. However, separation artifacts often distort the very Fx characteristics being studied, leading to significant performance drops. Fx-Encoder++ addresses this by "hearing through" complex musical mixtures to extract instrument-specific Fx representations directly, eliminating the need for imperfect source separation.
Methodology
Fx-Encoder++ is built on a contrastive learning framework (specifically SimCLR) and introduces a two-stage architecture designed for targeted extraction:
- Base Encoder: A PANN-based architecture that processes the complete mixture to create a high-level "mixture-level" embedding.
- Learnable Extractor: A 3-layer MLP that takes the mixture embedding and an instrument query to produce an instrument-specific Fx embedding.
To optimize performance, the model utilizes a pretrained CLAP model for queries, allowing users to specify instruments via text (e.g., "the sound of drums") or audio clips. Training strategies include Fx-normalization to isolate applied effects from recording environments and a dual-objective loss function that balances mixture-level and instrument-level contrastive tasks.
Experimental Setup
The model was trained on MoisesDB, featuring 240 multitrack recordings across 12 genres. Evaluation was conducted using a comprehensive benchmark on MUSDB18 and MedleyDB. To ensure robust ground truth, the researchers used synthetic augmentation via dasp-pytorch and torchcomp, applying random chains of seven effect types—including EQ, distortion, compression, and reverb—to create diverse production signatures.
Results and Insights
Fx-Encoder++ was evaluated on Audio Effects Retrieval and Parameter Matching tasks, demonstrating significant improvements over existing state-of-the-art (SOTA) models.
Mixture-Level and Instrument-Wise Performance
On the MUSDB18 mixture retrieval task, Fx-Encoder++ achieved a Recall@10 of 40.5%, outperforming previous models like FX-Encoder (34.5%) and AFx-Rep (36.9%). More importantly, in the novel task of instrument-wise extraction from mixtures, the model achieved a Recall@10 of 20.5% using audio queries. This nearly doubles the performance of baselines relying on universal source separation (11.5%).
Complexity as a Feature
A key insight from the study is that the model's performance actually improves as effect chains become more complex. Distinctive, "heavy" production creates a unique signature that is easier for the model to identify than subtle, single-effect processing.
Query Modalities
Interestingly, text queries occasionally outperformed audio queries. This suggests that text provides a more generalized representation of an instrument class, whereas audio queries might introduce distracting Fx or content from the query clip itself.
Conclusion and Future Work
Fx-Encoder++ proves that it is possible to extract specific instrument processing data from a mixture without the bottleneck of source separation. While the model excels at retrieval, future work will focus on improving the precision of numerical parameter matching and identifying very subtle, single-effect processing. The ultimate goal is to bridge the gap between representing an effect and applying it, potentially enabling automated "style transfer" for professional music production.
Summary updated