Neural scaling laws have revolutionized our understanding of Large Language Models (LLMs), yet their application to general audio representation learning remains challenging due to the multifactorial nature of audio data and training objectives. This paper introduces Embedding Effective Rank (RankMe) as a unifying metric to predict the performance of audio models across diverse variables such as data volume, model size, and masking strategies. We demonstrate that RankMe follows a consistent power-law relationship with downstream task performance, providing a label-free method for early-stage model evaluation and selection.
The Challenge of Audio Scaling
In domains like natural language processing and computer vision, scaling laws provide a predictable framework for model performance based on parameters, data, and compute. However, audio representation learning is often "messy," influenced by non-scalable variables like masking rates and architectural configurations that exhibit non-monotonic behavior. This complexity makes traditional scaling formulas difficult to apply directly to guide the development of audio foundation models.
The RankMe Framework
To address this, we propose using RankMe, an information-theoretic metric that calculates the effective rank of an embedding matrix via singular value decomposition. A high RankMe value indicates that a model utilizes its representational space efficiently, while a low value suggests representation collapse.
By mapping the impact of heterogeneous variables onto a single, predictable curve, we propose a new unified scaling law formula:
$$Q(R) = -(R_C/R)^{\alpha_R} + Q_\infty$$
where $Q$ represents quality (HEAR score) and $R$ is the RankMe value. This formulation allows researchers to evaluate the "scalability potential" of different architectures before committing to massive training runs.
Experimental Setup
Models were primarily based on the Dasheng Masked Autoencoding (MAE) architecture, though findings were validated across SSAST, HuBERT, and Wav2Vec2. Pre-training utilized a massive dataset of 100 million 10-second clips (~277,000 hours), combining AudioSet and ACAV100M. Evaluation was conducted on the HEAR (Holistic Evaluation of Audio Representations) benchmark, covering 16 tasks across speech, environmental sounds, and music.
Key Insights
RankMe as a Complexity Absorber
One of the most significant findings is RankMe's ability to linearize complex variables. For instance, while masking rates show non-monotonic trends when plotted directly against performance, they follow a clear power law when mapped through RankMe. RankMe effectively "compresses" multiple heterogeneous factors into one interpretable number.
Early-Bird Prediction
We observed a strong correlation (Pearson up to 0.86) between RankMe values at early training stages (100k–300k steps) and final downstream performance at 700k steps. This enables researchers to "pre-screen" model configurations early, significantly reducing computational waste by killing off underperforming runs.
The Scaling Frontier and Saturation
Similar to findings in NLP, we observed a "computational efficiency frontier" where larger models reach higher effective ranks with fewer total FLOPs than smaller models trained for longer. However, the theoretical maximum performance ($Q_\infty$) was consistently less than 1.0, suggesting that scaling current architectures alone may not lead to perfect representations without improvements in data quality or learning paradigms.
Conclusion
By unifying disparate variables into a single metric, RankMe provides a robust foundation for audio scaling laws. While saturation limits suggest that scaling alone has its bounds, RankMe offers a powerful, label-free tool for navigating the complex landscape of audio foundation models and predicting final quality from the earliest stages of training.
Summary updated