Why Larger Models Require Less Training Data

38 concepts31 papers1 proposal

A long-standing assumption in machine learning is that larger models require proportionally larger datasets to avoid overfitting. Under this traditional view, expanding a model's parameter count without a corresponding expansion in training data leads to memorization rather than generalization.

However, empirical evidence across vision, language, and recommendation systems reveals a counterintuitive reality: increasing model capacity inherently increases sample efficiency. Larger neural architectures extract more information from each individual training example, allowing them to reach target performance thresholds with significantly less data and fewer optimization steps than smaller models.

The Parameter-Data Trade-Off

When compute or data is constrained, the most effective strategy is often to scale up the model architecture while keeping the dataset fixed or even reducing it. This phenomenon manifests as a predictable trade-off between parameter scale and data density.

As Zhai et al. demonstrated in Scaling Vision Transformers, increased model complexity allows for a direct reduction in the total volume of supervised data required to achieve high performance. This sample-efficiency advantage is not unique to transformers; it spans diverse domains:

  • Sequential Recommendation: Larger recommendation models achieve comparable or superior performance with less user interaction data than smaller baselines, proving highly effective at capturing long-tail item dynamics (Zhang et al.).
  • Scene Text Recognition: Scaling model capacity to the billion-parameter range acts as a primary lever for reducing error rates independently of data growth, maximizing the utility of a fixed set of training samples (Rang et al.).
  • Multimodal Learning: In contrastive language-image pre-training, Li et al. identified an inverse scaling relationship in An Inverse Scaling Law for CLIP Training. They showed that larger encoders require significantly fewer image and text tokens during training to maintain high downstream performance.

Why Scale Accelerates Learning

Why does adding parameters make a model more data-efficient? The answer lies in how overparameterization alters the optimization landscape and enhances the model's capacity to form abstract representations.

1. Rapid Acquisition of Abstract Priors

Larger models possess a stronger intrinsic capacity to capture high-level structural regularities from minimal instruction. For example, Jaimovitch-Lopez et al. evaluated how language models internalize algorithmic patterns in Think Big, Teach Small: Do Language Models Distil Occam's Razor?. They found a significant correlation between parameter scale and a preference for simple algorithmic concepts. By distilling a mathematical simplicity prior directly from their architecture and optimization path, larger models require fewer examples to induce a task successfully.

2. Enhanced Signal Extraction per Token

A larger model extracts a stronger, cleaner semantic signal from sparse or low-fidelity inputs that would leave a smaller model underfitted. While processing a larger parameter set increases the computational cost per step, the model's learning capacity per training example increases even faster. The authors of Train Big, Then Compress note that this expanded capacity allows massive architectures to outperform smaller ones in wall-clock time by extracting more information from each training batch.

The "Train Big, Stop Early" Strategy

Because larger models learn more effectively per optimization step, they exhibit faster convergence rates. This leads to a highly practical training paradigm: train an oversized model for a limited time, stop early, and compress it for deployment.

Traditional Approach:
[Small Model] ───────────────────────────────► Full Convergence (Slow, lower accuracy)

Scale-Driven Approach:
[Massive Model] ──► Early Stop (Fast, high accuracy) ──► [Compression] ──► Efficient Deployment

In How to Train BERT with an Academic Budget, Izsak demonstrated that training a high-capacity BERT-Large architecture for a limited number of steps outperforms training a smaller BERT-Base model to full convergence within the same wall-clock time budget.

This accelerated convergence behavior is consistent across modalities. In specialized medical text processing, the authors of A large language model for electronic health records observed that higher model capacity optimizes the training process by requiring fewer passes through clinical text. Similarly, in generative modeling, Gao et al. showed in Lumina-T2X that large flow-based diffusion transformers reach convergence with a fraction of the total training compute required by smaller, previous-generation architectures.

The Limits of Scale: Low-Data Collapse

While scale drives data efficiency, this relationship is not entirely linear. In extreme low-data regimes, standard neural scaling laws can break down entirely.

As Wang et al. detailed in Data Efficient Neural Scaling Law via Model Reusing, when training data falls below a critical threshold relative to parameter count, larger models can experience a collapse in performance, exhibiting higher error rates than smaller counterparts.

This happens because massive models require a baseline volume of data to effectively adapt and coordinate their parameters. In severe few-shot scenarios, the high parameter count becomes an optimization liability rather than an asset (Li).

To prevent this low-data collapse, researchers rely on strategic weight initializations. By utilizing "model reusing"—initializing a massive model with weights from smaller, pre-trained architectures—the power-law scaling relationship is restored (Wang et al.). This allows large models to bypass the initial adaptation threshold and safely leverage their sample-efficiency advantages, even when training data is exceptionally scarce.

Key Foundational Work

Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020) — contributed 3 concept elements to this direction.
Factorization tricks for LSTM networks (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scale_dependent_data_efficiency_4e2f8b2c,
  title = {scale-dependent data efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-data-efficiency/rdr_4e2f8b2c6d964d6f90b6dd1b2b9eeb3b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scale_dependent_data_efficiency_4e2f8b2c_v1,
  title = {scale-dependent data efficiency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-dependent-data-efficiency/rdr_4e2f8b2c6d964d6f90b6dd1b2b9eeb3b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake