Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank

Xuyao Deng, Yanjie Sun, Yong Dou, Kele XuarXiv 2025 · DOI · arXiv

Neural scaling laws have revolutionized our understanding of Large Language Models (LLMs), yet their application to general audio representation learning remains challenging due to the multifactorial nature of audio data and training objectives. This paper introduces Embedding Effective Rank (RankMe) as a unifying metric to predict the performance of audio models across diverse variables such as data volume, model size, and masking strategies. We demonstrate that RankMe follows a consistent power-law relationship with downstream task performance, providing a label-free method for early-stage model evaluation and selection.

The Challenge of Audio Scaling

In domains like natural language processing and computer vision, scaling laws provide a predictable framework for model performance based on parameters, data, and compute. However, audio representation learning is often "messy," influenced by non-scalable variables like masking rates and architectural configurations that exhibit non-monotonic behavior. This complexity makes traditional scaling formulas difficult to apply directly to guide the development of audio foundation models.

The RankMe Framework

To address this, we propose using RankMe, an information-theoretic metric that calculates the effective rank of an embedding matrix via singular value decomposition. A high RankMe value indicates that a model utilizes its representational space efficiently, while a low value suggests representation collapse.

Figure 1: (a, b) The power-law relationship between RankMe and HEAR scores across different model scales. (c) Conceptual framework for unifying diverse variables through RankMe.

By mapping the impact of heterogeneous variables onto a single, predictable curve, we propose a new unified scaling law formula:

$$Q(R) = -(R_C/R)^{\alpha_R} + Q_\infty$$

where $Q$ represents quality (HEAR score) and $R$ is the RankMe value. This formulation allows researchers to evaluate the "scalability potential" of different architectures before committing to massive training runs.

Experimental Setup

Models were primarily based on the Dasheng Masked Autoencoding (MAE) architecture, though findings were validated across SSAST, HuBERT, and Wav2Vec2. Pre-training utilized a massive dataset of 100 million 10-second clips (~277,000 hours), combining AudioSet and ACAV100M. Evaluation was conducted on the HEAR (Holistic Evaluation of Audio Representations) benchmark, covering 16 tasks across speech, environmental sounds, and music.

Key Insights

RankMe as a Complexity Absorber

One of the most significant findings is RankMe's ability to linearize complex variables. For instance, while masking rates show non-monotonic trends when plotted directly against performance, they follow a clear power law when mapped through RankMe. RankMe effectively "compresses" multiple heterogeneous factors into one interpretable number.

Figure 4: Demonstration of RankMe as a complexity absorber. While masking rate (a, b) shows non-monotonic behavior relative to performance and rank, the relationship between RankMe and performance (c) follows a clear, unified power law.

Early-Bird Prediction

We observed a strong correlation (Pearson up to 0.86) between RankMe values at early training stages (100k–300k steps) and final downstream performance at 700k steps. This enables researchers to "pre-screen" model configurations early, significantly reducing computational waste by killing off underperforming runs.

The Scaling Frontier and Saturation

Similar to findings in NLP, we observed a "computational efficiency frontier" where larger models reach higher effective ranks with fewer total FLOPs than smaller models trained for longer. However, the theoretical maximum performance ($Q_\infty$) was consistently less than 1.0, suggesting that scaling current architectures alone may not lead to perfect representations without improvements in data quality or learning paradigms.

Conclusion

By unifying disparate variables into a single metric, RankMe provides a robust foundation for audio scaling laws. While saturation limits suggest that scaling alone has its bounds, RankMe offers a powerful, label-free tool for navigating the complex landscape of audio foundation models and predicting final quality from the earliest stages of training.

Summary updated

Abstract

Scaling laws have profoundly shaped our understanding of model performance in computer vision and natural language processing, yet their application to general audio representation learning remains underexplored. A key challenge lies in the multifactorial nature of general audio representation-representation quality is jointly influenced by variables such as audio length, embedding dimensionality, model depth, model architecture, data volume, etc., many of which are difficult to isolate or express analytically. In this work, we present a systematic study of scaling laws for general audio representations by utilizing embedding effective rank (RankMe) as a unifying metric that encapsulates the impact of diverse variables on representation quality. RankMe enables a label-free, information-theoretic quantification of audio embeddings, allowing us to examine scaling behaviors across a wide hyper-parameter space, including model size, training data volume, computational budget, architectural configurations, etc. Our empirical findings reveal a consistent power-law relationship between RankMe and representation quality, suggesting that embedding effective rank serves as a reliable proxy for assessing and predicting model performance in audio representation learning. This work not only validates the applicability of classical scaling principles to the general audio domain but also offers a theoretically grounded and empirically robust framework for guiding future model scaling strategies in audio foundation models.
BibTeX
@misc{deng2025unify,
  author = {Xuyao Deng and Yanjie Sun and Yong Dou and Kele Xu},
  title = {{Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank}},
  year = {2025},
  eprint = {2510.10948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10948},
}
Figure