TL;DR
While traditional deep learning scaling laws focus on dataset size and compute budget, scientific data is heavily bottlenecked by measurement noise. This paper introduces a universal, information-theoretic noise-scaling law that accurately predicts how biological representation learning models perform under varying noise levels. By evaluating 1,670 models, the authors demonstrate how this law can optimize experimental design and reveal critical trade-offs between model capacity and noise robustness.
What Problem Does This Paper Address?
In fields like computer vision and natural language processing, deep learning performance follows predictable "scaling laws" tied to data volume and compute. However, in biology, data quality is often severely limited by technical noise—such as molecular undersampling in single-cell RNA sequencing (scRNA-seq) or optical resolution limits in microscopy.
Until now, we lacked a systematic, quantitative framework to understand how measurement noise impacts a model's ability to learn meaningful biological representations. This paper addresses this gap, establishing noise as a distinct, predictable axis of model performance alongside data size and compute.
Methods
To measure representation quality across different architectures and noise levels without relying on incomparable loss functions, the authors introduced Auxiliary Mutual Information (MI). This technique generalizes standard "linear probing" by using a neural network-based estimator to calculate the mutual information between a model's learned representations ($Z$) and hidden, high-dimensional auxiliary biological signals ($Y$).
By modeling noise propagation as a Gaussian noise channel, the authors derived an analytical noise-scaling law:
$$I(\eta) = I_{\max} - \frac{1}{2} \log \frac{\eta/\bar{\eta} + 1}{\eta/\bar{\eta} + 2^{-2I_{\max}}}$$
Where:
- $\eta$ is the Signal-to-Noise Ratio (SNR).
- $I_{\max}$ represents Model Capacity (the maximum information extractable from a noiseless measurement).
- $\bar{\eta}$ represents Noise Sensitivity (a metric of a model's effective noise tolerance).
The authors validated this law by training 1,670 models across systematically degraded versions of genomic, imaging, and sequence datasets.
Datasets
The framework was tested across three distinct data modalities:
- Single-Cell & Spatial Transcriptomics: Mouse development scRNA-seq (~10⁷ cells), human peripheral blood mononuclear cells via CITE-seq (~10⁵ cells), mouse hematopoietic stem cells via LARRY lineage-traced scRNA-seq (~10⁵ cells), and a coronal mouse brain section via MERFISH spatial transcriptomics (~10⁵ cells).
- Images: A 5-class subset of the Caltech101 object recognition benchmark and a fluorescence microscopy dataset of human kidney cortex cells from MedMNIST v2 (~236,000 cells).
- Protein Sequences: ~63,000 SARS-CoV-2 spike protein sequences from the GISAID database.
Key Insights and Ideas
- A Universal Scaling Curve: Remarkably, when 193 distinct empirical noise-scaling curves across transcriptomics, imaging, and protein sequences were rescaled by their fitted capacity ($I_{\max}$) and noise sensitivity ($\bar{\eta}$), they collapsed perfectly onto a single universal theoretical curve.
- The Robustness vs. Capacity Trade-off: The authors uncovered a striking architectural trade-off in transcriptomics. The Transformer-based model (Geneformer) proved highly robust to noise, maintaining stable performance even at low sequencing depths. However, its ultimate capacity ($I_{\max}$) was significantly lower than the Variational Autoencoder (scVI), meaning scVI captures far more complex biological information when clean data is available.
- Intrinsic Modality Differences: Protein language models (like ESM2) are highly noise-robust and can largely ignore typical sequencing errors. Conversely, single-cell transcriptomics models are highly sensitive to noise, meaning current atlases remain strictly sensitivity-limited.
Key Results
- The proposed logarithmic scaling law fits empirical data exceptionally well, achieving an average $R^2 = 0.979 \pm 0.003$.
- Nonlinear models (Geneformer and scVI) showed 2.5- to 12.8-fold greater noise robustness than linear baselines like PCA.
- scVI consistently demonstrated the highest saturating performance, outperforming Geneformer's capacity ($I_{\max}$) by 0.4 to 1.4 bits across all tasks—effectively capturing double the signal complexity.
- Actionable Experimental Design: By inverting the scaling law, the authors calculated the exact sequencing depth ($UMI_{90}$) required for a model to hit 90% of its maximum performance. For instance, Geneformer operates near its asymptotic limit under typical sequencing depths, whereas scVI would benefit substantially from deeper sequencing on spatial and clonal tasks.
Limitations and Future Work
- Theoretical Origin: The derived law is exact for scalar Gaussian channels, yet it fits high-dimensional, highly non-Gaussian biological data surprisingly well. Explaining the theoretical underpinning of this generalization remains an open question.
- Architectural Scope: Due to the massive compute costs of training foundation models from scratch, only a few architectures (scVI, Geneformer, MobileNetV2, ESM2) were evaluated.
- Unifying the Scaling Axes: This study treats dataset size and measurement noise as independent variables. The authors note that a critical next step is developing a unified scaling law that simultaneously accounts for data volume, compute budget, model parameters, and noise levels.