Abstract
Pre-trained audio foundation models, such as CLAP and PANNs, have become standard feature extractors for various audio analysis tasks. However, the invariance properties of these embeddings—specifically their sensitivity to common audio effects like reverberation and filtering—are not well-characterized. This paper investigates how these effects warp the embedding space of three popular models: OpenL3, PANNs, and CLAP. Our analysis reveals that while audio effects move embeddings along predictable, monotonic paths, these "deformation trajectories" are high-dimensional and vary significantly between individual samples. This sample-specificity makes it challenging to "clean" embeddings using standard linear projections, as demonstrated by negligible performance gains in downstream instrument classification tasks.
Introduction
The rise of audio foundation models has shifted the paradigm of audio machine learning toward using pre-trained embeddings as input for downstream classifiers. For these embeddings to be robust, they should ideally exhibit invariance to "nuisance" parameters—incidental audio characteristics that do not change the semantic content. For instance, an embedding representing a "cello" should remain stable regardless of the recording environment's reverberation or the gain level. If embeddings are overly sensitive to these effects, classifiers trained on "clean" data may fail when deployed on "effected" real-world audio. This study systematically explores the nature of these deformations and evaluates potential mitigation strategies.
Methods
The researchers evaluated three models—OpenL3, PANNs, and CLAP—applying four parameterized effects to 3-second music excerpts from the IRMAS dataset (6,705 samples across 11 instrument classes):
- Gain: Volume adjustment.
- Low-pass filtering: High-frequency attenuation.
- Reverberation: Simulated room acoustics.
- Bitcrushing: Digital resolution reduction.
To characterize the impact of these effects, the study employed UMAP for visualization, Canonical Correlation Analysis (CCA) to identify effect-correlated directions, and Singular Value Decomposition (SVD) to assess the dimensionality of the resulting deformations. Finally, they attempted to improve classification robustness using linear subspace projections.
Analysis of Deformation Trajectories
Visualizing the embedding space using UMAP reveals distinct "trajectories" as effect intensity increases. As shown in Figure 1, while the movement is often monotonic—meaning the model effectively "tracks" the intensity of the effect—the paths taken by different samples are not parallel.
This visualization highlights a core challenge: the "deformation" is not a global shift that can be easily subtracted. Instead, the direction of movement in the high-dimensional space depends on the original audio content, a phenomenon the authors describe as the "local" vs. "global" problem. This suggests that foundation models do not globally linearize these effects.
Key Insights and Results
The study found that the correlation between embedding movement and effect strength is remarkably high. Global CCA yielded $R^2$ coefficients often exceeding 0.95 for OpenL3 and CLAP across most effects, indicating that these models are highly sensitive to effect parameters. However, SVD analysis indicated that these deformations are not confined to a few "nuisance dimensions"; the deformation subspace is nearly as high-dimensional as the original embedding space itself.
The practical impact of this sensitivity was tested through instrument classification. The researchers attempted to "debias" the embeddings by projecting out effect-sensitive dimensions before classification.
As illustrated in Figure 2, these linear mitigation techniques provided only marginal improvements (typically 0.003 to 0.01 ROC AUC) and, in some cases, even degraded performance. This confirms that the "damage" caused by audio effects is deeply integrated into the high-dimensional representation and cannot be rectified by simple linear subtractions.
Conclusion
Pre-trained audio embeddings are highly sensitive to common signal processing effects. Because these deformations are both high-dimensional and sample-specific, they pose a significant hurdle for simple post-processing robustness techniques. The findings suggest that linear projections are too blunt a tool for this task. Future research should focus on non-linear desensitization methods or the integration of these effects into the training process via data augmentation to foster true model invariance.
Summary updated