Abstract
We propose a method to automatically generate domain- and task-adaptive maskings of text for self-supervised pre-training, enabling effective language model (LM) adaptation to specific target tasks. We present a novel reinforcement learning-based framework, the Neural Mask Generator (NMG), which learns a masking policy such that the generated masks improve task performance on unseen texts during further pre-training. Using an off-policy actor-critic method with entropy regularization and a Transformer-based policy network, NMG considers the relative importance of words in context. We validate NMG on several question answering and text classification datasets using BERT and DistilBERT, demonstrating that it outperforms rule-based masking strategies by learning optimal adaptive maskings.
Introduction
Large-scale Language Models (LMs) like BERT are typically pre-trained on general-purpose corpora using a Masked Language Modeling (MLM) objective, where a random 15% of tokens are hidden. While effective for general language understanding, this "one-size-fits-all" approach often falls short in specialized domains like medicine or law.
The standard remedy is domain adaptation—performing "further pre-training" on domain-specific text. However, random masking is inefficient for this purpose. In specialized contexts, masking a common stopword like "the" provides negligible information compared to masking a critical medical entity or a task-relevant verb. The Neural Mask Generator (NMG) addresses this by treating mask selection as a meta-learning problem, automatically identifying which words are most "informative" for a specific task or domain.
Methodology
The NMG framework operates via a bi-level optimization process solved through Reinforcement Learning (RL).
- The Policy Network: The NMG is a Transformer-based network that takes contextualized embeddings of a sentence and outputs a probability distribution over potential masks.
- The RL Loop: The framework functions in two loops. In the inner loop, the target LM (e.g., DistilBERT) is pre-trained using masks from the NMG and then fine-tuned on a target task subset. In the outer loop, the NMG receives a reward based on the LM's performance improvement on a validation set, updating its policy to produce superior masks in subsequent iterations.
- Optimization Techniques: To stabilize training and handle discrete mask selection, the authors utilize off-policy actor-critic methods, entropy regularization, and "Self-Play," where the NMG competes against random baselines and its own previous versions.
Experimental Evaluation
The authors evaluated NMG across diverse Natural Language Understanding (NLU) tasks, including Question Answering (SQuAD, NewsQA, emrQA) and Text Classification (IMDb, ChemProt). NMG was compared against several heuristic baselines: Random Masking, Whole-Word Masking, Span Masking, and Entity Masking.
Key Results
NMG consistently outperformed or matched the best heuristic strategies across all datasets. In the medical domain (emrQA), NMG achieved an Exact Match (EM) score of 71.84, surpassing both random (71.40) and entity-based (71.50) masking. Similar gains were observed in the news domain and chemical relation extraction.
Analysis and Insights
Qualitative analysis reveals that NMG naturally learns to prioritize content over syntax. It ignores punctuation and functional words, focusing instead on nouns, verbs, and proper nouns. In QA tasks specifically, NMG learns to mask words that frequently appear in answer spans, forcing the model to learn the specific contextual bridges required for that domain. This targeted approach allows NMG to achieve superior adaptation results with fewer training steps than exhaustive random masking.
Conclusion and Future Work
While NMG demonstrates significant performance gains, it introduces computational overhead due to the RL training loop. Future research may explore more efficient RL algorithms to improve stability and investigate the use of NMG for generating adversarial masks to enhance model robustness. Despite these challenges, NMG represents a significant step toward automated, task-aware model adaptation that moves beyond simple heuristics.
Summary updated