Abstract
Large Pretrained Models (LPMs) are powerful but often sensitive to random seeds, leading to inconsistent performance. We present a two-stage fine-tuning framework that diagnoses model stability and enhances performance through variance reduction. By decomposing variance into sampling and optimization components, we identify a "Phase-II" of generalization where models are robust to training randomness. We then employ a dynamic snapshot ensembling (ENS) approach to minimize optimization noise. Our method yields substantial gains, especially in low-resource and imbalanced data scenarios like clinical NLP.
Introduction
In modern deep learning, massive models like BERT and RoBERTa often defy classical statistical expectations. While traditional theory suggests a U-shaped error curve where complexity eventually leads to overfitting, these over-parameterized models often exhibit "double descent," performing better as they grow. However, they remain notoriously "brittle." A model might achieve state-of-the-art (SOTA) results with one random seed but fail with another, a phenomenon often called the "optimization lottery." This paper, presented at ACL 2023, proposes a mathematically grounded framework to diagnose this brittleness and systematically improve model robustness.
Methodology
Stage-I: Justification of Generalization
The authors decompose the model's error into three distinct components:
- Bias: The inherent error of the model architecture.
- Variance due to Sampling ($E_{varS}$): Error stemming from the specific luck of the data split.
- Variance due to Optimization ($E_{varO}$): Error resulting from the luck of the random seed or initialization.
Based on this, the framework defines two phases of training. Phase-I occurs when $E_{varO} > E_{varS}$, suggesting the model has not yet converged to a stable solution and is dominated by optimization noise. The goal is to reach Phase-II, where $E_{varO} \leq E_{varS}$, indicating the model is robust to training randomness.
Stage-II: Ensembling to Reduce Variance
Once a model is identified as being in a specific phase, the authors apply ENS (Dynamic Snapshot Ensemble). Instead of a single training run, the method saves multiple "snapshots" of the model during training. A pruning algorithm then selects the most effective subset of these snapshots to form an ensemble. This process specifically targets and reduces $E_{varO}$ (optimization noise) while maintaining a fixed bias, effectively stabilizing the model's predictions.
Experimental Setup and Datasets
The framework was evaluated on a diverse set of benchmarks:
- SuperGLUE: Includes tasks like BoolQ (QA), CB (inference), and COPA (reasoning).
- Clinical NLP (THYME and i2b2): High-stakes datasets involving temporal relation extraction from medical records, characterized by extreme class imbalance.
Key Results and Insights
SOTA Models and Phase-I
Surprisingly, many current SOTA configurations for tasks like MultiRC or COPA were found to still be in Phase-I. This indicates they are highly sensitive to initialization and that significant performance gains remain "unclaimed" due to optimization noise.
Performance Gains
The two-stage method consistently outperformed standard Replicated SOTA (RSOTA) settings. Notably, on the CB (CommitmentBank) task—a small dataset—the method achieved a 49.3% relative F1 improvement. In clinical settings, the ensemble significantly boosted performance for rare, low-resource classes.
Ensembling vs. Scaling
While the industry trend is to scale models to reduce bias, this research suggests that increasing ensemble size ($N$) is a more computationally efficient way to reduce variance. Ensembling acts as a filter, stabilizing weak signals that are often lost in the noise of learning rare patterns.
Conclusion
This work provides a rigorous path for moving beyond the "optimization lottery" in large language models. By diagnosing the phase of generalization and applying targeted variance reduction through snapshot ensembling, researchers can achieve more robust and higher-performing models, particularly in challenging, low-resource domains. Future work will look toward distilling these robust ensembles back into single, efficient models.
Summary updated