Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures

Yen-Tung Yeh, Junghyun Koo, Marco A. Martínez Ramírez, Wei-Hsiang Liao, Yi-Hsuan Yang, Yuki Mitsufuji

TL;DR

Fx-Encoder++ is the first audio representation model capable of extracting instrument-specific audio effects (Fx) directly from complete music mixtures without needing source separation. By combining a SimCLR contrastive learning framework with a queryable "extractor" powered by CLAP embeddings, it allows users to condition on an instrument via text or audio and isolate how effects were applied to that specific stem.

What Problem Does This Paper Address?

In intelligent music production (e.g., automated mixing and mastering), algorithms must understand how individual stems—like vocals, drums, or bass—are processed within a song. However, existing audio representations exhibit two critical bottlenecks:

  1. General-purpose audio encoders (like CLAP or PANN) prioritize semantic content recognition over the subtle timbral transformations induced by effects like compression, reverb, or equalization.
  2. Prior effect-specific encoders (such as FX-Encoder or AFx-Rep) only analyze effects applied to isolated tracks or summarize the aggregate mixture as a whole.

A naive workaround—isolating an instrument using source separation before analyzing its effects—fails because source separation algorithms introduce phase smearing, transient blurring, and high-frequency loss that warp the effect's signature. Fx-Encoder++ bridges this gap by directly disentangling instrument-specific effects from mixed audio.

Methods

Fx-Encoder++ adopts a two-stage contrastive architecture:

  • Mixture-Level Representation: A PANN-based CNN processes log-mel spectrograms of audio mixtures into global effect embeddings. To keep the model from learning "shortcuts" based on musical pitch or timbre, the authors enforce:
  • Fx-Normalization: Preprocessing that neutralizes pre-existing audio effects in stems.
  • Consistent Instrumentation: Ensuring positive and negative samples within a batch contain identical instrument combinations.
  • Hard Negative Mining: Audio with identical musical content processed with structurally different effect chains.
  • Learnable Extractor Mechanism: Instead of conditioning the backbone directly (which complicates global representation learning), a 3-layer MLP extractor takes the mixture embedding alongside an instrument query embedding $Q$ generated by a pretrained CLAP model: $$z^m = \text{extractor}(Q^m, \mathcal{E}(M))$$ Queries can be either reference audio or text prompts (e.g., "this is the sound of {instrument}").

  • Dual Contrastive Objective & Curriculum: The model optimizes both a mixture loss $\ell_{\text{mixture}}$ and an instrument-conditioned triplet loss $\ell_{\text{inst}}$. Training begins with 100% mixture loss and linearly ramps to $\lambda_{\text{mix}} = 0.8$ and $\lambda_{\text{inst}} = 0.2$, giving the network time to establish robust holistic representations before isolating stem components.

       [ Audio Mixture ] ──> [ PANN Encoder ] ──> [ Mixture Latent ]
                                                        │
[ CLAP Query (Audio/Text) ] ────────────────────> [ Extractor MLP ] ──> [ Instrument-Wise Latent ]

Datasets

  • Training: MoisesDB (240 multitrack songs across 12 genres, >14 hours of data, utilizing up to 11 stem types per track).
  • Evaluation Datasets:
  • MUSDB18-HQ: Standard four-stem evaluation (vocals, drums, bass, other).
  • MedleyDB: Out-of-distribution evaluation for instrument generalization (mandolin, saxophone, horn, trumpet).
  • Benchmarking Suite: An expanded retrieval task with 500 candidate configurations (compared to 20 in prior work) and randomized effect chains ranging from 1 to 8 cascaded processors (EQ, compressor, limiter, reverb, distortion, delay, stereo imager).

Key Insights and Ideas

  • Separation Artifacts Ruin Effects Analysis: When testing a separate-then-analyze baseline using music source separation (Demucs), retrieval recall dropped by ~19% compared to ground-truth clean stems. This confirms that separation artifacts directly degrade effect signatures.
  • Text Queries Outperform Audio Queries: When querying the extractor, text prompts ("this is the sound of drums") yielded slightly higher accuracy than actual audio clips. Text queries provide an invariant, abstract representation of the instrument category, whereas reference audio queries carry unrelated acoustic variance and effects that confuse the extractor.
  • Complexity Helps Representation: Retrieval performance scales positively with the number of cascaded effects (1 through 8). A single effect is easily masked by natural instrument timbre, while complex effect chains impart a distinct, identifiable signature.

Key Results

  • Mixture-Level Retrieval: Fx-Encoder++ outperforms previous state-of-the-art effect models on MUSDB18 mixtures, achieving 40.5% Recall@10 (compared to AFx-Rep’s 36.9% and FX-Encoder’s 34.5%).
  • Instrument-Wise Retrieval from Mixtures: When extracting instrument effects straight from a mix without separation:
  • On MUSDB18, text-conditioned extraction achieved 12.2% Recall@10 (5.3% mAP@10), outperforming universal source separation pipelines (USS Recall@10: 9.9%).
  • Downstream Parameter Matching: In optimizing differentiable effect chain parameters to match target audio, Fx-Encoder++ achieves competitive spectral loss ($L_d$), particularly on bass and out-of-distribution brass/strings, though results vary across different instrument categories.

Limitations and Future Work

  • Single-Effect Disentanglement: The model struggles to differentiate a single isolated effect from native instrument timbre, performing best when multiple effects are chained.
  • Parameter Matching Gap: While Fx-Encoder++ leads in retrieval, parameter estimation tasks yielded mixed results—in some cases, general representations like VGGish performed comparably, indicating that retrieval-optimized representations do not seamlessly translate to parametric optimization.
  • Unseen Instruments: Performance noticeably drops on MedleyDB’s acoustic instruments (horn, trumpet, mandolin), revealing sensitivity to instruments that were sparse in the MoisesDB training set.

Summary updated

Abstract

General-purpose audio representations have proven effective across diverse music information retrieval applications, yet their utility in intelligent music production remains limited by insufficient understanding of audio effects (Fx). Although previous approaches have emphasized audio effects analysis at the mixture level, this focus falls short for tasks demanding instrument-wise audio effects understanding, such as automatic mixing. In this work, we present Fx-Encoder++, a novel model designed to extract instrument-wise audio effects representations from music mixtures. Our approach leverages a contrastive learning framework and introduces an ``extractor'' mechanism that, when provided with instrument queries (audio or text), transforms mixture-level audio effect embeddings into instrument-wise audio effect embeddings. We evaluated our model across retrieval and audio effect parameter matching tasks, testing its performance across a diverse range of instruments. The results demonstrate that Fx-Encoder++ outperforms previous approaches at mixture level and show a novel ability to extract effects representation instrument-wise, addressing a critical capability gap in intelligent music production systems.
BibTeX
@inproceedings{yeh2025fx,
  author = {Yen-Tung Yeh and Junghyun Koo and Marco A. Martínez Ramírez and Wei-Hsiang Liao and Yi-Hsuan Yang and Yuki Mitsufuji},
  title = {{Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures}},
  booktitle = {ISMIR},
  year = {2025},
  url = {https://openreview.net/forum?id=uFyNxU8nq4},
}

Loading related content...

Figure