When fine-tuning a massive pre-trained language model on a small dataset, a frustrating phenomenon frequently occurs: running the exact same code with a different random seed can cause the final performance to swing wildly between state-of-the-art accuracy and random guessing. This high sensitivity to initial conditions is often called the "initialization lottery" (Supervised Contrastive Learning Approach for Contextual Ranking).
In data-starved environments, fine-tuning becomes highly volatile. Understanding why this instability happens—and how to mitigate it—requires looking closely at the under-constrained optimization landscapes of large models.
The Mechanics of Fine-Tuning Volatility
Why does a simple change in a random seed or data shuffling order cause such drastic performance fluctuations?
The core issue is that small datasets fail to adequately constrain the optimization process of over-parameterized models. As Ruiqi Zhong et al. demonstrated in Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level, the stochasticity introduced during the task-specific fine-tuning phase (such as data shuffling and classification head initialization) contributes two to eight times more to prediction variance than the randomness in the initial pre-training phase.
When data is scarce, this stochasticity manifests in three distinct ways:
- Suboptimal Optimization Defaults: Standard optimization choices that work well on large datasets can cause catastrophic failures on small ones. Marius Mosbach in On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines and Tianyi Zhang 0007 et al. in Revisiting Few-sample BERT Fine-tuning identified that common fine-tuning instability is often driven by optimization oversights, such as omitting Adam's bias correction, poor top-layer initialization, and training for too few iterations, which together lead to vanishing gradients and early divergence.
- Representational Collapse: As a model forces its massive parameter space to fit a handful of examples, it often discards the rich, general-purpose features it acquired during pre-training. Armen Aghajanyan documented this in Better Fine-Tuning by Reducing Representational Collapse, showing that without explicit grounding, the model's internal representations degrade rapidly.
- Layer-Wise Quality Imbalance: In low-data regimes, the internal layers of a network do not update uniformly. Zihang Liu et al. showed in Model Balancing Helps Low-data Training and Fine-tuning that limited data causes a significant drift in the internal state of the network, where some layers over-index on the training samples while others barely adapt at all.
Because of these factors, the model's optimization trajectory becomes highly sensitive to its starting point, leading to unpredictable downstream results (Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts).
Constraining the Parameter Space
To combat this instability, researchers have developed methods to anchor the model, preventing it from drifting into fragile local minima.
One prominent strategy is to restrict the number of parameters allowed to change. By keeping the bulk of the pre-trained weights frozen and updating only a targeted subset, the optimization landscape is drastically simplified. For instance, Bo Pang et al. demonstrated in Robust Transfer Learning with Pretrained Language Models through Adapters that using small, trainable adapter layers isolates the tuning process and insulates the model from initialization sensitivity.
Similarly, limiting updates to specific subnetworks can stabilize few-shot learning. Runxin Xu et al. showed in S$^4$-Tuning: A Simple Cross-lingual Sub-network Tuning Method that restricting parameter updates to language-specific sub-networks significantly reduces the standard deviation of performance across different training runs. This concept is expanded by Haojie Zhang et al. in Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively, which adaptively selects promising subnetworks during training to prevent overfitting and ensure consistent convergence.
Standard Fine-Tuning (Unconstrained)
[Pre-trained Weights] ---> (All Parameters Updated) ---> High Variance / Collapse
Parameter-Constrained Fine-Tuning
[Pre-trained Weights (Frozen)] + [Adapters/Subnetworks (Tuned)] ---> Stable Convergence
Another approach focuses on regularizing the representations themselves. Instead of freezing weights, methods like the one proposed by Anastasia Razdaibiedina et al. in Representation Projection Invariance Mitigates Representation Collapse force fine-tuned representations to remain mathematically recoverable to their original pre-trained states. This keeps the model grounded in its general-purpose knowledge base, making the fine-tuning process far less dependent on random initialization seeds.
Task Reformulation and Smoothing
Beyond modifying parameters and architectures, stability can be achieved by changing how the task is presented to the model or how the optimization path is smoothed.
Prompting as an Optimization Anchor
Initializing a brand-new classification head introduces a massive injection of random weights that must be learned from scratch. Prompt-based tuning avoids this entirely by reformulating the downstream task as a language modeling problem.
Maosong Sun et al. showed in Prompt Tuning for Discriminative Pre-trained Language Models that prompt-based task reformulations effectively mitigate the high variance and training divergence typically seen in standard fine-tuning. In fact, Alexander M Rush et al. discovered in How many data points is a prompt worth? that in low-data regimes, the variance introduced by random initialization seeds often completely dwarfs the variance caused by the choice of different human-written prompts. By leveraging the pre-trained head, prompting anchors the model to a well-behaved region of the loss landscape from day one.
Smoothing via Noise and Ensembles
If the underlying problem is a rugged, under-constrained loss landscape, injecting controlled stochasticity can paradoxically help. Branislav Pecher et al. introduced this intuition in Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation. By applying noisy interpolation and delayed ensembling, their approach smooths out the optimization trajectory, preventing the model from falling into the fragile, high-variance local minima that characterize low-data training failures.
Key Foundational Work
Semi-supervised Learning with Weakly-Related Unlabeled Data: Towards Better Text Categorization (2008) — contributed 1 concept elements to this direction.
What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment (2016) — contributed 1 concept elements to this direction.