Compute-Optimal Scaling Laws Beyond Autoregressive Text

This direction is merged into another direction: compute-optimal resource allocation. Its last version stays readable and citable here.

Version 1 of 123 papers23 conceptsfrozen 2026-10-04

In the early days of large-scale deep learning, the recipe for better performance was simple: make the model bigger. However, the "Chinchilla" scaling laws shifted the paradigm by showing that for a fixed compute budget, there is a precise mathematical "sweet spot" between the number of model parameters ($N$) and the number of training tokens ($D$).

Recent research has moved beyond standard autoregressive language models to determine if these predictable power laws hold for other architectures—like xLSTM and Diffusion—and other modalities—like pixels, proteins, and robot motion. The consensus is clear: while the specific "appetites" for data and parameters change, the underlying physics of scaling remains remarkably consistent.

The Engine of Discovery: IsoFLOP Profiling

To find a scaling law, researchers don't just train one massive model. Instead, they perform IsoFLOP profiling. This involves training dozens or hundreds of smaller models across varying compute budgets.

For any given budget (e.g., $10^{18}$ FLOPs), there is a trade-off: you can train a large model on a small amount of data, or a small model on a large amount of data. By plotting the final loss of these configurations, researchers find a "U-shaped" curve. The bottom of that U represents the compute-optimal configuration for that budget.

By connecting the bottoms of these U-curves across multiple orders of magnitude of compute, researchers derive a power law that predicts the performance of models far larger than any they have actually trained. This methodology has been applied to everything from Imitation Learning in NetHack by Dean Foster and colleagues to Large Recommendation Models by Kshitij Jain et al.

Different Tasks Have Different Appetites

One of the most significant findings in recent scaling research is that "intelligence" is not a monolithic metric. Different skills require different resource allocations.

Knowledge vs. Reasoning

Nicholas Roberts and Niladri S. Chatterji demonstrated in Compute Optimal Scaling of Skills: Knowledge vs Reasoning that knowledge-heavy tasks are capacity-hungry. To store more facts, you need more parameters. Conversely, reasoning-heavy tasks (like coding) are data-hungry; they benefit more from seeing a wider variety of logic patterns in the training data than from simply having a larger brain.

Pixels vs. Text

When moving from text to vision, the efficiency of scaling changes. Xinchen Yan and Chen Liang found that raw pixel prediction follows predictable trends but is roughly 10–20× less efficient than text. In Rethinking generative image pretraining, they show that image generation is "data-hungry," while image classification is "model-hungry."

Scaling New Architectures

Does the Transformer hold a monopoly on efficient scaling? Recent work suggests otherwise.

  • xLSTM: Maximilian Beck and Sepp Hochreiter established scaling laws for xLSTM, showing it consistently Pareto-dominates Transformers across five orders of magnitude of compute, offering lower loss for the same budget while maintaining linear time complexity.
  • Masked Diffusion: Qian Liu and Shen Nie proved that Masked Diffusion Models (MDMs) scale as efficiently as standard autoregressive models. However, Jinjie Ni et al. noted in their work on Quokka that while Diffusion Language Models scale predictably, they are significantly more data-hungry than their autoregressive counterparts.
  • Diffusion Transformers (DiT): In the visual domain, Zhengyang Liang and Hao He established the first explicit scaling laws for DiTs, providing a roadmap for balancing model size and data to maximize image generation quality.

Beyond Static Training

Scaling laws are also being used to make training itself more dynamic. Rather than keeping a model's architecture fixed, Sotiris Anagnostidis and Thomas Hofmann propose Adaptive Strategies. By using scaling laws to mathematically determine the most efficient path, they can change a model’s "shape"—such as patch size in Vision Transformers—during training. This allows the model to stay on the "steepest descent" path of the compute-error frontier, reducing training FLOPs by up to 60%.

Why It Works: The Predictability of Information

The reason these laws work across such diverse fields—from Protein Language Models by Bo Chen to Motion Forecasting for Self-Driving Cars by Kratarth Goel—is that they describe the fundamental relationship between model capacity (the ability to store patterns) and signal exposure (the opportunity to learn patterns).

By treating compute as a finite resource to be optimized, researchers can now forecast the performance of future systems. As Xinchen Yan predicts, these laws suggest that even "inefficient" methods like pixel-by-pixel modeling will become computationally competitive with specialized models within just a few years, simply by following the inevitable curve of the power law.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compute_optimal_scaling_laws_beyond_auto_0a31600c,
  title = {Compute-Optimal Scaling Laws Beyond Autoregressive Text},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compute-optimal-scaling-laws-beyond-autoregressive-text/rdr_0a31600c22d34ecdb08196fe6d186f11},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_compute_optimal_scaling_laws_beyond_auto_0a31600c_v1,
  title = {Compute-Optimal Scaling Laws Beyond Autoregressive Text},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compute-optimal-scaling-laws-beyond-autoregressive-text/rdr_0a31600c22d34ecdb08196fe6d186f11/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}