Cliff-Learning

Tony T. Wang, Igor Zablotchi, Nir Shavit, Jonathan S. RosenfeldarXiv 2023 · DOI · arXiv

Abstract

Traditional scaling laws in deep learning predict error reduction as a power law of data volume. However, recent research identifies a "cliff-learning" phenomenon in foundation models, where error drops precipitously in low-data regimes. This paper explores the mechanisms behind these cliffs, demonstrating that high compatibility between pre-trained priors and downstream tasks enables hyper-efficient learning, often reaching human-level performance with 1000x less data than training from scratch.

Introduction

In the era of foundation models, the standard paradigm of "more data equals better performance" is being refined. While large-scale training follows predictable power laws, the adaptation of these models to specific tasks often exhibits non-linear, highly efficient behavior. This "cliff-learning" suggests that the "return on data" is not constant but can be dramatically accelerated when a model's internal representations align with the target task.

The Scaling Law Departure

Scaling laws typically follow a power law: $E \propto n^{-\beta}$, appearing as a straight line on a log-log plot. This paper addresses the "low-data" regime in transfer learning—finetuning a massive pre-trained model for a specific task. In these scenarios, the straight line breaks. The error drops like falling off a cliff before eventually settling back into a standard power law slope.

Cliff-learning in CLIP compared to traditional scaling and data-pruning baselines.

Methodology

The researchers combined empirical studies on state-of-the-art models with mathematical toy models to isolate the cause of the cliff. They evaluated various versions of OpenAI’s CLIP and LAION-CLIP models on standard benchmarks like CIFAR-10, CIFAR-100, and SVHN.

Key adaptation methods compared include:

  • Linear Probing: Training only the final classification layer.
  • Full Finetuning: Updating all model weights.
  • k-Nearest Neighbors: A non-parametric approach to classification.
Comparison of adaptation methods: linear probing vs. finetuning across different model architectures.

Key Insights

  • Compatibility as a Catalyst: A "cliff" represents a "lightbulb moment" where the model's internal priors perfectly align with the task. If a foundation model already "understands" the basic features, it only needs a handful of examples to "click" into place.
  • Log-Log Concavity: Mathematically, cliff-learning is defined by concavity on a log-log plot. While power laws represent the limit of expected learning, cliff-learning represents a period of hyper-efficiency where the return on data is at its highest.
  • Quality and Aggression: Higher-quality foundation models don't just start with lower error; they "cliff" more aggressively, reaching high performance with orders of magnitude less downstream data.

Key Results

  • 1000x Data Efficiency: Finetuning a CLIP model on CIFAR-10 was found to be 1000 times more data-efficient than training a model from scratch to reach human-level performance.
  • The Critical Threshold: In linear regression toy models, the researchers identified a sharp cliff where error drops to zero exactly when the number of training samples ($n$) equals the number of dimensions ($d$).
  • Shared Asymptotics: Interestingly, while different finetuning methods have different "cliffs," they often converge to the same power-law slope once they have sufficient data. The cliff effectively determines the "head start" a model receives.

Discussion and Limitations

Cliff-learning is not universal; for instance, models may require full finetuning rather than just a linear probe to exhibit cliff-like behavior on certain datasets like SVHN. A major open question is whether it's possible to achieve cliff-learning when training from scratch, or if a massive amount of data must always be consumed somewhere in the pipeline. Additionally, some experiments showed a non-monotonic "bump" in error post-cliff, suggesting complex underlying learning dynamics that warrant further investigation.

Summary updated

Abstract

We study the data-scaling of transfer learning from foundation models in the low-downstream-data regime. We observe an intriguing phenomenon which we call cliff-learning. Cliff-learning refers to regions of data-scaling laws where performance improves at a faster than power law rate (i.e. regions of concavity on a log-log scaling plot). We conduct an in-depth investigation of foundation-model cliff-learning and study toy models of the phenomenon. We observe that the degree of cliff-learning reflects the degree of compatibility between the priors of a learning algorithm and the task being learned.
BibTeX
@misc{wang2023cliff,
  author = {Tony T. Wang and Igor Zablotchi and Nir Shavit and Jonathan S. Rosenfeld},
  title = {{Cliff-Learning}},
  year = {2023},
  eprint = {2302.07348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.07348},
}
Figure