Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models

Lijing Wang, Yingya Li, Timothy Miller, Steven Bethard, Guergana SavovaACL 2023 · DOI

Abstract

Large Pretrained Models (LPMs) are powerful but often sensitive to random seeds, leading to inconsistent performance. We present a two-stage fine-tuning framework that diagnoses model stability and enhances performance through variance reduction. By decomposing variance into sampling and optimization components, we identify a "Phase-II" of generalization where models are robust to training randomness. We then employ a dynamic snapshot ensembling (ENS) approach to minimize optimization noise. Our method yields substantial gains, especially in low-resource and imbalanced data scenarios like clinical NLP.

Introduction

In modern deep learning, massive models like BERT and RoBERTa often defy classical statistical expectations. While traditional theory suggests a U-shaped error curve where complexity eventually leads to overfitting, these over-parameterized models often exhibit "double descent," performing better as they grow. However, they remain notoriously "brittle." A model might achieve state-of-the-art (SOTA) results with one random seed but fail with another, a phenomenon often called the "optimization lottery." This paper, presented at ACL 2023, proposes a mathematically grounded framework to diagnose this brittleness and systematically improve model robustness.

Methodology

Stage-I: Justification of Generalization

The authors decompose the model's error into three distinct components:

  1. Bias: The inherent error of the model architecture.
  2. Variance due to Sampling ($E_{varS}$): Error stemming from the specific luck of the data split.
  3. Variance due to Optimization ($E_{varO}$): Error resulting from the luck of the random seed or initialization.
Figure 1: Decomposition of bias and variance across model complexity, illustrating the transition from Phase-I to Phase-II at the cross point where optimization variance meets sampling variance.

Based on this, the framework defines two phases of training. Phase-I occurs when $E_{varO} > E_{varS}$, suggesting the model has not yet converged to a stable solution and is dominated by optimization noise. The goal is to reach Phase-II, where $E_{varO} \leq E_{varS}$, indicating the model is robust to training randomness.

Stage-II: Ensembling to Reduce Variance

Once a model is identified as being in a specific phase, the authors apply ENS (Dynamic Snapshot Ensemble). Instead of a single training run, the method saves multiple "snapshots" of the model during training. A pruning algorithm then selects the most effective subset of these snapshots to form an ensemble. This process specifically targets and reduces $E_{varO}$ (optimization noise) while maintaining a fixed bias, effectively stabilizing the model's predictions.

Experimental Setup and Datasets

The framework was evaluated on a diverse set of benchmarks:

  • SuperGLUE: Includes tasks like BoolQ (QA), CB (inference), and COPA (reasoning).
  • Clinical NLP (THYME and i2b2): High-stakes datasets involving temporal relation extraction from medical records, characterized by extreme class imbalance.

Key Results and Insights

SOTA Models and Phase-I

Surprisingly, many current SOTA configurations for tasks like MultiRC or COPA were found to still be in Phase-I. This indicates they are highly sensitive to initialization and that significant performance gains remain "unclaimed" due to optimization noise.

Performance Gains

The two-stage method consistently outperformed standard Replicated SOTA (RSOTA) settings. Notably, on the CB (CommitmentBank) task—a small dataset—the method achieved a 49.3% relative F1 improvement. In clinical settings, the ensemble significantly boosted performance for rare, low-resource classes.

Figure 6: Performance comparison between standard SOTA (RSOTA) and the proposed ensemble method (ENS) across various datasets, showing consistent improvements (IPV) in Precision, Recall, and F1 scores.

Ensembling vs. Scaling

While the industry trend is to scale models to reduce bias, this research suggests that increasing ensemble size ($N$) is a more computationally efficient way to reduce variance. Ensembling acts as a filter, stabilizing weak signals that are often lost in the noise of learning rare patterns.

Conclusion

This work provides a rigorous path for moving beyond the "optimization lottery" in large language models. By diagnosing the phase of generalization and applying targeted variance reduction through snapshot ensembling, researchers can achieve more robust and higher-performing models, particularly in challenging, low-resource domains. Future work will look toward distilling these robust ensembles back into single, efficient models.

Summary updated

BibTeX
@inproceedings{wang2023two,
  author = {Lijing Wang and Yingya Li and Timothy Miller and Steven Bethard and Guergana Savova},
  title = {{Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.877},
}
Figure