Abstract
We propose a method to automatically generate domain- and task-adaptive maskings of text for self-supervised pre-training, effectively adapting language models to specific target tasks like question answering. We present a novel reinforcement learning-based framework, the Neural Mask Generator (NMG), which learns a masking policy to maximize task performance on unseen texts. Using off-policy actor-critic with entropy regularization and experience replay, we develop a Transformer-based policy network that evaluates word importance. Validated on question answering and text classification datasets using BERT and DistilBERT, NMG outperforms rule-based strategies by learning optimal adaptive maskings.
Introduction
The standard paradigm for training Large Language Models (LLMs) like BERT relies heavily on Masked Language Modeling (MLM), where a fixed percentage (typically 15%) of tokens are masked at random for the model to predict. While effective for general-purpose pre-training, this "one-size-fits-all" approach is inefficient for domain-specific adaptation. In specialized fields such as medicine or law, masking common stop words provides little learning signal compared to masking domain-critical entities or relational verbs.
Existing solutions often rely on manual heuristics, such as Whole Word Masking or Entity Masking. However, these rules are static and fail to account for the specific requirements of the downstream task. To address this, we introduce the Neural Mask Generator (NMG), a framework that treats masking as a meta-learning problem, automatically discovering which tokens are most "informative" to hide for a given task.
Methodology: The Neural Mask Generator
The NMG framework operates through a bi-level optimization process. At its core, it consists of a policy network that assigns masking probabilities to tokens based on their context.
Bi-level Architecture
The system functions in two distinct loops:
- Inner Loop: A target language model (e.g., BERT) undergoes further pre-training and fine-tuning using the masks generated by the NMG.
- Outer Loop: The NMG observes the performance of the fine-tuned model on a validation set. This performance serves as a "reward" signal, which is used to update the NMG’s masking policy via reinforcement learning.
Reinforcement Learning Strategy
Directly calculating gradients through the entire training process of a large model is computationally prohibitive. To overcome this, we employ off-policy actor-critic reinforcement learning with entropy regularization. The NMG acts as the "agent," the masking decisions are "actions," and the improvement in task accuracy is the "reward." To ensure training stability, we implement a self-play mechanism where the NMG competes against its previous iterations and random baselines.
Experimental Setup
We evaluated NMG across several diverse benchmarks:
- Question Answering (QA): SQuAD v1.1 (general), NewsQA (news), and emrQA (medical).
- Text Classification: IMDb (sentiment) and ChemProt (chemical-protein interactions).
The experiments utilized BERT-base and DistilBERT as the underlying language models to test the framework's effectiveness across different model scales.
Key Results and Analysis
NMG consistently outperformed traditional heuristic masking strategies across nearly all tested datasets and base models.
Performance Gains in Specialized Domains
The benefits of adaptive masking were most pronounced in specialized domains. For instance, on the medical-themed emrQA dataset, NMG achieved significant improvements in both Exact Match (EM) and F1 scores compared to random or rule-based masking. This suggests that NMG successfully identifies and masks complex medical terminology that is crucial for the model to understand the domain.
Qualitative Insights
Analysis of the generated masks revealed that NMG prioritizes "content" words—specifically nouns and verbs—over functional stop words or punctuation. Interestingly, the optimal strategy was found to be model-dependent; the NMG learned different masking patterns for DistilBERT than for BERT, suggesting that the "difficulty" of the task must be calibrated to the capacity of the model being adapted.
Conclusion
The Neural Mask Generator represents a shift from static, human-designed masking rules toward dynamic, learned adaptation strategies. By leveraging reinforcement learning to identify task-relevant tokens, NMG provides a more efficient and effective path for domain adaptation. Future work will focus on reducing the computational overhead of the RL training process and exploring the potential for zero-shot transfer of learned masking policies across related domains.
Summary updated