Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms

Ankit Shah, Shuyi Chen, Kejun Zhou, Yue Chen, Bhiksha RajarXiv 2023 · DOI · arXiv

Introduction

The pursuit of a "universal" audio representation—a single model capable of interpreting diverse acoustic signals from musical performances to environmental noise—remains a central challenge in audio AI. This paper investigates the factors contributing to model generalization, specifically within the context of the Holistic Evaluation of Audio Representations (HEAR) 2021 challenge. The authors propose Batch Embedding Covariance Regularization (BECR), a novel technique designed to encourage "dispersed" embeddings, and evaluate the efficacy of biologically inspired preprocessing methods like the Constant-Q Transform (CQT).

Methodology: BECR and the PaSST Architecture

The research utilizes the Patchout Transformer (PaSST) as its foundational architecture. To enhance generalization, the authors introduce BECR, a supplementary loss term applied during training.

Proposed training and evaluation framework incorporating BECR.

BECR leverages the Gini Index—a metric traditionally used in economics to measure wealth inequality—to quantify "information inequality" within embedding spaces. By calculating the covariance matrix of batch embeddings and applying the Gini Index to its eigenvalues, the model is incentivized to distribute information more evenly across all dimensions. To ensure computational efficiency, the authors utilize a trace-based optimization, reducing complexity from $O(K^3)$ to $O(K^2)$, making the regularization virtually cost-free in terms of training time.

Additionally, the study compares standard Short-time Fourier Transform (STFT) preprocessing with the Constant-Q Transform (CQT). While CQT's logarithmic frequency spacing mimics human auditory perception, its compatibility with modern transformer architectures like PaSST was previously untested in this context.

Key Insights: The Value of Dispersion

A critical finding of this research is the strong correlation between embedding dispersion and downstream task performance. Models that achieved higher scores across diverse datasets (FSD50K, NSynth, CREMA-D) consistently exhibited embeddings where information was not concentrated in a few dominant principal components.

Correlation between performance and the Gini Index of normalized eigenvalues across different datasets.

As shown in the analysis of the Gini Index, higher equality in eigenvalue distribution (representing more dispersed information) directly relates to improved model performance. This confirms that forcing a model to utilize its entire embedding space prevents it from over-relying on specific features that may not generalize well.

Results and Discussion

The experimental results demonstrate the clear benefits of BECR. On the FSD50K dataset, the inclusion of BECR raised the mean Average Precision (mAP) from a baseline of 27.8% to 36.8%. Similar improvements were observed across speech and music tasks.

Conversely, the experiment with CQT yielded surprising results. Despite its biological plausibility, CQT-based models significantly underperformed compared to STFT. For instance, on the NSynth Pitch task, the STFT+BECR configuration reached 51.2% accuracy, whereas the CQT variant achieved only 4.8%. This suggests that current state-of-the-art architectures may be implicitly optimized for the linear frequency resolution provided by STFT.

Conclusion

This paper establishes BECR as an effective, computationally efficient method for improving the generalization of audio representations. By using the Gini Index to promote embedding dispersion, models can better handle the variety of tasks required for holistic audio understanding. While STFT remains the superior preprocessing choice for PaSST-like models, future research may explore whether alternative architectures can better leverage the unique properties of CQT.

Summary updated

Abstract

General-purpose embedding is highly desirable for few-shot even zero-shot learning in many application scenarios, including audio tasks. In order to understand representations better, we conducted a thorough error analysis and visualization of HEAR 2021 submission results. Inspired by the analysis, this work experiments with different front-end audio preprocessing methods, including Constant-Q Transform (CQT) and Short-time Fourier transform (STFT), and proposes a Batch Embedding Covariance Regularization (BECR) term to uncover a more holistic simulation of the frequency information received by the human auditory system. We tested the models on the suite of HEAR 2021 tasks, which encompass a broad category of tasks. Preliminary results show (1) the proposed BECR can incur a more dispersed embedding on the test set, (2) BECR improves the PaSST model without extra computation complexity, and (3) STFT preprocessing outperforms CQT in all tasks we tested. Github:https://github.com/ankitshah009/general_audio_embedding_hear_2021
BibTeX
@misc{shah2023approach,
  author = {Ankit Shah and Shuyi Chen and Kejun Zhou and Yue Chen and Bhiksha Raj},
  title = {{Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms}},
  year = {2023},
  eprint = {2303.03591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.03591},
}
Figure