Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation

Minki Kang, Moonsu Han, Sung Ju Hwang

Abstract

We propose a method to automatically generate domain- and task-adaptive maskings of text for self-supervised pre-training, enabling effective language model (LM) adaptation to specific target tasks. We present a novel reinforcement learning-based framework, the Neural Mask Generator (NMG), which learns a masking policy such that the generated masks improve task performance on unseen texts during further pre-training. Using an off-policy actor-critic method with entropy regularization and a Transformer-based policy network, NMG considers the relative importance of words in context. We validate NMG on several question answering and text classification datasets using BERT and DistilBERT, demonstrating that it outperforms rule-based masking strategies by learning optimal adaptive maskings.

Introduction

Large-scale Language Models (LMs) like BERT are typically pre-trained on general-purpose corpora using a Masked Language Modeling (MLM) objective, where a random 15% of tokens are hidden. While effective for general language understanding, this "one-size-fits-all" approach often falls short in specialized domains like medicine or law.

The standard remedy is domain adaptation—performing "further pre-training" on domain-specific text. However, random masking is inefficient for this purpose. In specialized contexts, masking a common stopword like "the" provides negligible information compared to masking a critical medical entity or a task-relevant verb. The Neural Mask Generator (NMG) addresses this by treating mask selection as a meta-learning problem, automatically identifying which words are most "informative" for a specific task or domain.

Methodology

The NMG framework operates via a bi-level optimization process solved through Reinforcement Learning (RL).

  1. The Policy Network: The NMG is a Transformer-based network that takes contextualized embeddings of a sentence and outputs a probability distribution over potential masks.
  2. The RL Loop: The framework functions in two loops. In the inner loop, the target LM (e.g., DistilBERT) is pre-trained using masks from the NMG and then fine-tuned on a target task subset. In the outer loop, the NMG receives a reward based on the LM's performance improvement on a validation set, updating its policy to produce superior masks in subsequent iterations.
  3. Optimization Techniques: To stabilize training and handle discrete mask selection, the authors utilize off-policy actor-critic methods, entropy regularization, and "Self-Play," where the NMG competes against random baselines and its own previous versions.
The NMG framework uses reinforcement learning to optimize a masking policy based on downstream task performance.

Experimental Evaluation

The authors evaluated NMG across diverse Natural Language Understanding (NLU) tasks, including Question Answering (SQuAD, NewsQA, emrQA) and Text Classification (IMDb, ChemProt). NMG was compared against several heuristic baselines: Random Masking, Whole-Word Masking, Span Masking, and Entity Masking.

Key Results

NMG consistently outperformed or matched the best heuristic strategies across all datasets. In the medical domain (emrQA), NMG achieved an Exact Match (EM) score of 71.84, surpassing both random (71.40) and entity-based (71.50) masking. Similar gains were observed in the news domain and chemical relation extraction.

Performance comparison of NMG against heuristic masking strategies on various Question Answering datasets.

Analysis and Insights

Qualitative analysis reveals that NMG naturally learns to prioritize content over syntax. It ignores punctuation and functional words, focusing instead on nouns, verbs, and proper nouns. In QA tasks specifically, NMG learns to mask words that frequently appear in answer spans, forcing the model to learn the specific contextual bridges required for that domain. This targeted approach allows NMG to achieve superior adaptation results with fewer training steps than exhaustive random masking.

Conclusion and Future Work

While NMG demonstrates significant performance gains, it introduces computational overhead due to the RL training loop. Future research may explore more efficient RL algorithms to improve stability and investigate the use of NMG for generating adversarial masks to enhance model robustness. Despite these challenges, NMG represents a significant step toward automated, task-aware model adaptation that moves beyond simple heuristics.

Summary updated

Abstract

We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.
BibTeX
@inproceedings{kang2020neural,
  author = {Minki Kang and Moonsu Han and Sung Ju Hwang},
  title = {{Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OLANU_A_BKM},
}
Figure