The art and principle of universal scaling
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Scaling laws have become the bedrock for foundation model training, yet standard scaling laws are largely restricted to evaluating cross-entropy loss at terminal checkpoints -- leaving the continuous dynamics of general observables under-explored. In this work, we establish universal scaling laws across large language models (LLM) and multi-modal models (MMLM) that accurately characterize the any-time-any-scale trajectories of losses, evaluation metrics, and parameter norms. To establish this unified framework, we integrate three key pillars: conventional end-point scaling laws, convex optimization theory, and an enhanced collapse transformation. We systematically validate this framework across diverse model architectures, optimizers, training hyperparameters, compute regimes, and modalities, providing a principled recipe for universal scaling.
BibTeX
@misc{bu2027art,
author = {Zhiqi Bu and Junlin Han and Zeyuan Allen-Zhu and Ashok Cutkosky and Aaron Defazio and Mike Lewis and Lin Xiao},
title = {{The art and principle of universal scaling}},
year = {2027},
url = {https://openreview.net/forum?id=VKQoV053dT},
howpublished = {OpenReview},
note = {Submitted to ICLR 2027},
}Loading related content...