Abstract
Pre-trained models have become indispensable for efficiently building models across a broad spectrum of downstream tasks. The advantages of pre-trained models have been highlighted by empirical studies on scaling laws, which demonstrate that larger pre-trained models can significantly reduce the sample complexity of downstream learning. However, existing theoretical investigations of pre-trained models lack the capability to explain this phenomenon. In this paper, we provide a theoretical investigation by introducing a novel framework, caulking, inspired by parameter-efficient fine-tuning (PEFT) methods such as adapter-based fine-tuning, low-rank adaptation, and partial fine-tuning. Our analysis establishes that improved pre-trained models provably decrease the sample complexity of downstream tasks, thereby offering theoretical justification for the empirically observed scaling laws relating pre-trained model size to downstream performance, a relationship not covered by existing results.
Introduction
In modern deep learning, "Scaling Laws" serve as a foundational observation: as the size of a pre-trained model ($m$) increases, the amount of target data ($n$) required to achieve a certain performance level on a downstream task decreases. Despite this empirical consistency, standard statistical learning theory has struggled to provide a rigorous explanation.
Traditional error rates are typically expressed as a sum of independent terms, such as $m^{-\alpha} + n^{-\beta}$. In this formulation, even an infinitely large pre-trained model ($m \to \infty$) leaves the error bottlenecked by the inherent complexity of the target task ($n^{-\beta}$), where $\beta$ is a fixed constant. This paper addresses this theoretical gap by demonstrating that $\beta$ is not fixed; rather, larger models fundamentally simplify the target task, making it easier to learn with fewer samples.
The Caulking Framework
To bridge this gap, the authors introduce a novel theoretical framework termed "caulking," a name inspired by the process of sealing gaps. This framework is modeled after Parameter-Efficient Fine-Tuning (PEFT) techniques like Adapters and Low-Rank Adaptation (LoRA).
The framework conceptualizes a pre-trained model as a composition of a feature extractor ($g_e$) and a head ($g_h$). Caulking involves inserting a task-specific adapter ($g_a$) between these two components. The researchers define caulkability as the property where a target function can be accurately reconstructed by inserting a simple adapter into a pre-trained base. By applying empirical caulking—a least-squares estimator restricted to this specific architecture—the authors analyze error bounds through the lens of nonparametric regression, specifically utilizing $\beta$-complexity and Hölder continuity.
Theoretical Insights: The "Simpler Adapter" Hypothesis
The core insight of the paper is that as a pre-trained model scales, it doesn't just become more accurate; it becomes "wider" in its representational capacity. This increased breadth means the mathematical "gap" between the model's existing knowledge and the requirements of a new task becomes simpler to bridge.
In a multi-layer compositional space, a high-quality pre-trained model allows the learner to bypass the complexity of learning a feature extractor and head from scratch. The theory demonstrates that the influence of these complex components is effectively eliminated from the error bound, leaving only the complexity of the adapter. Consequently, if a larger model allows for a simpler adapter, the error rate on the target task improves from a fixed $n^{-\beta}$ to a more favorable $n^{-\beta + o(1)}$, where the $o(1)$ term represents the efficiency bonus gained from the superior pre-trained base.
Experimental Validation
The authors validated their theoretical findings across several benchmarks:
- CNN Scaling (Office-Home & ImageNet): Experiments using ResNet-50 and Wide ResNet-50-2 demonstrated that the larger Wide ResNet-50-2 achieved its peak performance with a single-layer adapter. In contrast, the smaller ResNet-50 required a more complex, multi-layer adapter to reach its (lower) performance ceiling.
- Vision-Language Models (MMStar): When building VLMs using SigLIP encoders, a larger 303M parameter encoder consistently outperformed an 86M parameter version, particularly when paired with simpler adapter architectures. This confirms that larger foundation components reduce the complexity of the integration task.
Conclusion and Future Work
This work provides the first rigorous mathematical justification for why methods like LoRA and Adapters are so effective: they exploit the inherent caulkability of large-scale models. While the paper proves that scaling improves sample complexity if a model is caulkable, it leaves open the question of how to guarantee caulkability during the pre-training phase. Future research directions include extending the framework to cover multi-layer modifications like LoRA and investigating scenarios where target tasks are significantly out-of-distribution relative to the pre-training data.
Summary updated