Exploring Post-Hoc Disentanglement of Audio Foundation Models for Music Similarity

Lucas de Souza Brandão, Gabriel da Costa Merlin, Diego Furtado SilvaarXiv 2026 · arXiv

TL;DR

This paper explores how to "untangle" the complex embedding spaces of frozen audio foundation models (like CLAP, MERT, and MuQ) to enable fine-grained music retrieval. By adding a lightweight adaptation layer and attribute-specific masks, the authors demonstrate that you can achieve high-quality, controllable similarity search (e.g., matching by "mood" while ignoring "genre") without the massive computational cost of retraining the backbone models from scratch.

What Problem Does This Paper Address?

Modern audio foundation models are excellent at capturing "general" musical similarity, but their internal representations are entangled. If you ask a model for a track similar to a jazz ballad, it might return another jazz track (matching genre) or another slow, melancholic track (matching mood). Currently, you can't easily tell the model to "find something with this specific mood, but I don't care about the genre."

While Conditional Similarity Networks (CSN) were designed to solve this by gating embeddings into attribute-specific subspaces, they are typically trained from scratch. This paper asks a critical question for the resource-constrained researcher: Can we achieve this disentanglement post-hoc by simply adapting the outputs of existing, frozen foundation models?

Methods

The authors propose a framework that sits on top of frozen audio backbones (CLAP, MuQ, and MERT). The pipeline consists of:

  1. Temporal Pooling: Reducing long audio sequences into fixed-size vectors using either simple Mean Pooling or a parametric Attention Pooling mechanism.
  2. Adaptation Modules: This is the "secret sauce." They test three levels of complexity: Identity (no change), a One Layer projection, and a multi-layer perceptron (MLP). These modules learn to reshape the foundation model's latent space.
  3. Masking Strategies: They apply attribute-specific masks (for Genre, Instrument, and Mood) to "gate" the embedding. They compare Fixed Masks (pre-defined, non-overlapping chunks of the vector) against Learned Masks.
  4. Objective Function: The system is trained using a triplet loss designed to pull similar tracks together in the masked subspace, combined with regularization terms to ensure mask sparsity and orthogonality (keeping the subspaces distinct).

Datasets

The study utilizes two primary music information retrieval (MIR) benchmarks:

  • MTG-Jamendo: Used for both training and evaluation, featuring over 55,000 tracks with rich tag metadata.
  • MagnaTagATune (MTT): Used strictly as a held-out test set to evaluate how well the post-hoc disentanglement generalizes to new, unseen data distributions.

Key Insights and Ideas

  • Adaptation is Mandatory: Simply applying masks to a frozen foundation model (the "Identity" approach) fails. You need at least a little bit of trainable "glue" (like a single linear layer) to reorganize the features before masking works effectively.
  • The "Suppression" Effect: Interestingly, the performance gains in disentanglement come primarily from suppressing interfering information from other attributes rather than "injecting" new signal. The masks act as filters that block out the "noise" of irrelevant musical facets.
  • Redundancy is Everywhere: The fact that Fixed Masks (which just grab a random 1/3 of the embedding dimensions) perform remarkably well suggests that attribute information is distributed redundantly throughout the foundation models' latent spaces.
  • Efficiency: Minimal adaptation capacity is sufficient. You don't need deep networks to fix the embeddings; a lightweight MLP is often enough to unlock significant interpretability.

Key Results

The authors found that adding adaptation modules led to substantial improvements in both retrieval accuracy and "Specificity" (a metric for how well a mask targets one attribute without affecting others).

Table 1: Macro-average performance across configurations. Note the significant jump in accuracy for MERT and MuQ when moving from 'Identity' to 'One Layer' or 'MLP' modules.

As shown in Table 1, the baseline performance of models like MERT and MuQ is significantly boosted by the post-hoc adaptation. Furthermore, Table 2 highlights that the Specificity score—crucial for true disentanglement—is highest when using trained adaptation modules.

Table 2: Average Specificity scores. Higher scores indicate better attribute isolation. The 'Identity' configurations (top rows) perform poorly compared to 'One Layer' and 'MLP' versions.

Finally, Table 3 illustrates the "push-pull" nature of disentanglement: applying a "Genre" mask increases genre retrieval accuracy while intentionally decreasing accuracy for "Instrument" and "Mood," proving the mask is successfully isolating the desired facet.

Table 3: Detailed performance of the MLP configuration. Arrows indicate the increase in target attribute performance and the corresponding decrease in non-target attributes.

Limitations and Future Work

  • Interference vs. Innovation: The study notes that masks mostly work by suppressing cross-attribute interference. Future work could explore if there are ways to actually enhance specific features that the foundation model might have under-represented.
  • Architectural Sensitivity: Results varied depending on the specific backbone and dataset, suggesting there isn't a "one-size-fits-all" configuration for post-hoc adaptation.
  • Scaling Attributes: This study focused on three broad categories (genre, instrument, mood). Scaling this to dozens of fine-grained attributes (tempo, key, era, etc.) remains a challenge for future research.

Summary updated

Abstract

Audio foundation models like CLAP, MuQ, and MERT excel at capturing general musical similarity. On the other hand, their entangled latent spaces limit interpretability and fine-grained attribute control. Conditional Similarity Networks (CSN) address this by pairing a backbone representation model with learned masks that gate the embedding space into attribute-specific subspaces. However, existing methods train the backbone from scratch, rather than leveraging pretrained foundation models. We instead present a systematic study of post-hoc disentanglement, adapting CSN to gate frozen pretrained embeddings. We evaluate adding trainable parameters to adapt the embedding under different masking and pooling regimes across two datasets. Our findings show that minimal adaptation capacity is sufficient for post-hoc gating to improve both attribute-specific disentanglement and overall retrieval performance, while pooling, masking, and depth choices yield limited additional gains.
BibTeX
@misc{brandao2026exploring,
  author = {Lucas de Souza Brandão and Gabriel da Costa Merlin and Diego Furtado Silva},
  title = {{Exploring Post-Hoc Disentanglement of Audio Foundation Models for Music Similarity}},
  year = {2026},
  eprint = {2611.14415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14415},
}
Figure