The art and principle of universal scaling

Zhiqi Bu, Junlin Han, Zeyuan Allen-Zhu, Ashok Cutkosky, Aaron Defazio, Mike Lewis, Lin XiaoOpenReview 2027 · Submission to ICLR 2027 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Scaling laws have become the bedrock for foundation model training, yet standard scaling laws are largely restricted to evaluating cross-entropy loss at terminal checkpoints -- leaving the continuous dynamics of general observables under-explored. In this work, we establish universal scaling laws across large language models (LLM) and multi-modal models (MMLM) that accurately characterize the any-time-any-scale trajectories of losses, evaluation metrics, and parameter norms. To establish this unified framework, we integrate three key pillars: conventional end-point scaling laws, convex optimization theory, and an enhanced collapse transformation. We systematically validate this framework across diverse model architectures, optimizers, training hyperparameters, compute regimes, and modalities, providing a principled recipe for universal scaling.
BibTeX
@misc{bu2027art,
  author = {Zhiqi Bu and Junlin Han and Zeyuan Allen-Zhu and Ashok Cutkosky and Aaron Defazio and Mike Lewis and Lin Xiao},
  title = {{The art and principle of universal scaling}},
  year = {2027},
  url = {https://openreview.net/forum?id=VKQoV053dT},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2027},
}

Loading related content...

Figure