Linear Scaling of Performance Under Distribution Shift

This direction is split into other directions: linear performance correlation. Its last version stays readable and citable here.

Version 1 of 152 papers72 conceptsfrozen 2026-10-04

When machine learning models encounter data that differs from their training set—a scenario known as distribution shift—they typically suffer a drop in accuracy. One might expect these failures to be chaotic or highly dependent on specific model architectures. However, a growing body of research reveals a surprising regularity: a model's out-of-distribution (OOD) performance is often a nearly perfect linear function of its in-distribution (ID) performance.

This phenomenon, frequently termed "Accuracy on the Line," suggests that the way models fail is not random, but follows a predictable trajectory governed by the strength of the underlying signal.

The Linear Correlation Phenomenon

The core observation, popularized by John Miller 0001, Aditi Raghunathan, and colleagues in Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization, is that if you plot the ID accuracy of many different models against their OOD accuracy on a specific shifted dataset, the points fall on a straight line.

This relationship holds across diverse architectures (CNNs, Transformers), training durations, and hyperparameters. It implies that for many real-world shifts, the most effective way to improve robustness is simply to build a more accurate base model. As Horia Mania and Suvrit Sra argue in Why do classifier accuracies show linear trends under distribution shift?, this trend emerges naturally if models are "ordered"—meaning a high-accuracy model generally correctly classifies everything a lower-accuracy model does, plus some harder examples.

Why the Line Exists: Probit Scaling

The linearity becomes even more precise when viewed through a "probit" lens. By transforming accuracy using the inverse of the Gaussian cumulative distribution function (the probit function), researchers have found that the relationship between ID and OOD performance reflects a consistent scaling of the signal-to-noise ratio.

Essentially, a distribution shift acts like a constant "noise" filter applied to the model's internal representations. If a model becomes better at extracting signal from the original data, that improvement translates proportionally to the shifted data, even if the absolute accuracy is lower. This suggests that current training paradigms produce models that fail in highly synchronized ways, as noted in OODRobustBench by Lin Li and collaborators.

Practical Utility: Agreement on the Line

This predictability offers a powerful tool for model deployment. In many real-world settings, we have plenty of unlabeled data from a new environment but no ground-truth labels to measure accuracy.

Mingjie Sun and J. Zico Kolter demonstrate in Reliable Test-Time Adaptation via Agreement-on-the-Line that we can estimate OOD accuracy by looking at the agreement between two different models. Because the correlation is so strong, the rate at which two models agree on a shifted dataset is linearly related to their actual accuracy on that dataset. This allows for label-free performance estimation and hyperparameter tuning during test-time adaptation.

When the Line Breaks

While the linear trend is robust, it is not universal. Identifying where it fails is key to understanding "true" robustness:

  1. Subpopulation Shifts: Weixin Liang and James Y. Zou show in Accuracy on the Curve that when shifts involve underrepresented groups or spurious correlations, the relationship can become nonlinear (a "moon shape"). In these cases, improving average ID accuracy might actually hurt performance on specific minority subgroups.
  2. Effective Robustness: Most models simply move along the line. A model is "effectively robust" only if it sits significantly above the line—performing better than its ID accuracy would predict. Sara Fridovich-Keil and colleagues use a Fourier Lens to show that models like CLIP achieve this by learning features that are more invariant to high-frequency noise.
  3. Benchmark Misspecification: As explored by Olawale Salaudeen in Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?, a perfect linear correlation can sometimes be a "red flag." It may indicate that a benchmark is too easy, allowing models to rely on the same shortcuts in both ID and OOD data rather than forcing them to learn generalizable features.

Implications for Research

The "Accuracy on the Line" research direction shifts the focus from simply measuring OOD drops to understanding the slope and intercept of these performance lines. It suggests that many specialized "robustness" interventions are actually just standard accuracy improvements in disguise. To move beyond this, researchers are now looking for methods—such as multicalibration (Jiayun Wu et al., Bridging Multicalibration and Out-of-distribution Generalization) or spectral filtering—that can fundamentally change the relationship between training success and real-world reliability.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_scaling_of_performance_under_dist_ff1baa02,
  title = {Linear Scaling of Performance Under Distribution Shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-scaling-of-performance-under-distribution-shift/rdr_ff1baa0294844130956e8b284747d5b1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_scaling_of_performance_under_dist_ff1baa02_v1,
  title = {Linear Scaling of Performance Under Distribution Shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-scaling-of-performance-under-distribution-shift/rdr_ff1baa0294844130956e8b284747d5b1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}