An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models

Xueqing Liu, Chi Wang

Fine-tuning pre-trained language models (PLMs) like RoBERTa and Electra is the standard approach for achieving state-of-the-art results in Natural Language Processing (NLP). However, model performance is highly sensitive to hyperparameter configurations. While automated Hyperparameter Optimization (HPO) tools promise to replace manual tuning, recent empirical evidence suggests they often fail to outperform simple grid search without specific interventions.

The HPO Paradox in Fine-Tuning

Automated HPO methods—such as Random Search (RS), Asynchronous Successive Halving (ASHA), and Bayesian Optimization (BO)—are designed to explore continuous search spaces more efficiently than rigid grid searches. Despite their theoretical advantages and native support in popular libraries like HuggingFace, their practical effectiveness in the context of fine-tuning PLMs has been inconsistent. This study investigates why these tools often struggle and proposes a systematic approach to improve their reliability.

Methodology and Troubleshooting

The researchers evaluated HPO performance using Electra-base and RoBERTa-base across the GLUE benchmark. They identified two primary failure modes for HPO: insufficient time budgets and severe overfitting. To address these, they introduced a systematic Troubleshooting Procedure.

The systematic troubleshooting procedure for HPO.

As shown in the flowchart above, the procedure begins with a standard time budget (1GST, or the time required for a standard grid search). If HPO fails to outperform the baseline, the budget is increased (e.g., to 4GST). If performance remains poor due to overfitting, the search space is reduced by fixing "risky" hyperparameters—such as warmup ratio and dropout—to default values, focusing the search only on learning rate and batch size.

Experimental Results

The study revealed several critical insights into the performance of HPO versus grid search. Initially, under equal time budgets, HPO failed to match grid search on several tasks, including RTE, STS-B, and QNLI for Electra. This is largely attributed to the "1GST Fallacy": because HPO explores a much larger, continuous space, it requires more trials to find optimal configurations than a discrete grid search.

Furthermore, HPO frequently fell into an "Overfitting Trap." On smaller datasets like WNLI and MRPC, HPO algorithms often found configurations that achieved near-perfect validation scores but generalized poorly to test sets.

Performance comparison of grid search and HPO methods on Electra-base.

As illustrated in the results for Electra-base, while HPO methods like BO+ASHA often achieved higher validation scores than grid search (e.g., on WNLI and CoLA), these gains did not always translate to the test set. For instance, on WNLI, BO+ASHA reached a validation score of 58.2 compared to grid search's 56.3, yet its test score (61.6) was significantly lower than the grid search baseline (65.1).

Key Insights and Recommendations

  1. Over-optimization of Validation Sets: HPO is highly effective at optimizing the target metric, but in low-resource fine-tuning scenarios, this often leads to picking hyperparameters (like low warmup ratios) that cause the model to memorize the validation set.
  2. ASHA's Aggressive Pruning: Early-stopping algorithms like ASHA can be too aggressive, pruning trials that start slowly but might have reached higher final performance.
  3. Efficiency of Troubleshooting: The proposed troubleshooting procedure proved more efficient than brute-force enumeration, saving approximately 60% in computational resources while eventually allowing HPO to outperform grid search in several challenging tasks like CoLA and MNLI.

Conclusion

Automated HPO is not a "set-and-forget" solution for fine-tuning PLMs. To succeed, practitioners must provide HPO with sufficient time budgets and be prepared to constrain the search space to prevent overfitting on small validation sets. Future work should focus on developing automated detection for overfitting during the HPO process and identifying validation metrics that better correlate with generalizable performance.

Summary updated

BibTeX
@inproceedings{liu2021empirical,
  author = {Xueqing Liu and Chi Wang},
  title = {{An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.178},
}
Figure