Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking

Iñigo Urteaga, Moulay-Zaïdane Draïdia, Tomer Lancewicki, Shahram KhadiviACL 2023 · DOI

Abstract: Training large-scale Transformer-based Language Models (TLMs) is computationally intensive, with hyperparameter selection often requiring expensive grid searches. This paper presents a Bayesian optimization framework utilizing Multi-Armed Bandits (MAB) to dynamically tune hyperparameters during the pre-training phase. By employing Gaussian Process-based Thompson Sampling (GP-TS), we demonstrate that masking strategies can be optimized on-the-fly, leading to significant reductions in training loss and improved downstream task performance with minimal computational overhead.

Introduction

The pre-training of Transformer-based models like BERT and RoBERTa has revolutionized natural language processing but at a high environmental and financial cost. A critical factor in training efficiency is the selection of hyperparameters, such as the masking probability in Masked Language Modeling (MLM). Traditional methods rely on static "best-guess" values or exhaustive grid searches, both of which are inefficient. This research proposes a dynamic approach: finding the optimal configuration sequentially as the model trains.

Methodology

We frame hyperparameter selection as a sequential decision problem. A Multi-Armed Bandit (MAB) agent interacts with the training loop, where each "arm" represents a specific masking probability. To navigate the exploration-exploitation trade-off, we utilize Gaussian Process-based Thompson Sampling (GP-TS).

  • Gaussian Process (GP): Acts as a surrogate model that predicts the "reward" (reduction in validation loss) for specific hyperparameter settings, including those not yet tested.
  • Thompson Sampling: A strategy that balances exploration (trying new settings) and exploitation (using known high-performing settings).

This allows the agent to adapt the training difficulty dynamically, maintaining the model in an optimal learning state throughout the pre-training process.

Experimental Evaluation

The framework was evaluated across three diverse datasets to prove its versatility:

  • wiki-c4: A standard general-purpose corpus combining Wikipedia and web crawl data.
  • mimic: Specialized clinical notes from the MIMIC-III database.
  • e-commerce: A private dataset of marketplace product titles and descriptions.

Downstream performance was assessed using Medical Natural Language Inference (MLI) and various e-commerce classification tasks (e.g., title quality and similarity).

Key Results and Discussion

The GP-TS approach consistently outperformed fixed-parameter baselines. As shown in the figure below, the dynamic agent achieves lower validation loss across all three domains more rapidly than static masking rates (represented by various $\rho$ values).

Validation loss convergence across wiki-c4, mimic, and e-commerce datasets comparing fixed masking probabilities against the dynamic GP-TS approach.

Key findings include:

  • Efficiency: GP-TS reached lower MLM loss in fewer training epochs. In the e-commerce domain, the model reached 98.0% accuracy in title classification within just 10 interactions.
  • Negligible Overhead: The computational cost of running the GP-TS agent is between 0.05% and 1% of total training time, making it a highly efficient alternative to full grid searches.
  • Continual Pre-training: The method is equally effective when fine-tuning general models (like RoBERTa) on specialized domains, finding optimal domain-specific strategies faster than human experts.

Conclusion

Dynamic hyperparameter tuning via multi-armed bandits offers a promising path toward more sustainable and efficient AI development. By optimizing masking strategies in real-time, researchers can achieve better-performing models with significantly fewer resources. Future work will explore extending this framework to causal language models (like GPT) and a broader range of hyperparameters beyond masking probabilities.

Summary updated

BibTeX
@inproceedings{urteaga2023multi,
  author = {Iñigo Urteaga and Moulay-Zaïdane Draïdia and Tomer Lancewicki and Shahram Khadivi},
  title = {{Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.675},
}
Figure