Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation

Minki Kang, Moonsu Han, Sung Ju Hwang

Abstract

We propose a method to automatically generate domain- and task-adaptive maskings of text for self-supervised pre-training, effectively adapting language models to specific target tasks like question answering. We present a novel reinforcement learning-based framework, the Neural Mask Generator (NMG), which learns a masking policy to maximize task performance on unseen texts. Using off-policy actor-critic with entropy regularization and experience replay, we develop a Transformer-based policy network that evaluates word importance. Validated on question answering and text classification datasets using BERT and DistilBERT, NMG outperforms rule-based strategies by learning optimal adaptive maskings.

Introduction

The standard paradigm for training Large Language Models (LLMs) like BERT relies heavily on Masked Language Modeling (MLM), where a fixed percentage (typically 15%) of tokens are masked at random for the model to predict. While effective for general-purpose pre-training, this "one-size-fits-all" approach is inefficient for domain-specific adaptation. In specialized fields such as medicine or law, masking common stop words provides little learning signal compared to masking domain-critical entities or relational verbs.

Existing solutions often rely on manual heuristics, such as Whole Word Masking or Entity Masking. However, these rules are static and fail to account for the specific requirements of the downstream task. To address this, we introduce the Neural Mask Generator (NMG), a framework that treats masking as a meta-learning problem, automatically discovering which tokens are most "informative" to hide for a given task.

Methodology: The Neural Mask Generator

The NMG framework operates through a bi-level optimization process. At its core, it consists of a policy network that assigns masking probabilities to tokens based on their context.

Bi-level Architecture

The system functions in two distinct loops:

  1. Inner Loop: A target language model (e.g., BERT) undergoes further pre-training and fine-tuning using the masks generated by the NMG.
  2. Outer Loop: The NMG observes the performance of the fine-tuned model on a validation set. This performance serves as a "reward" signal, which is used to update the NMG’s masking policy via reinforcement learning.
The NMG framework uses an RL-based approach where a mask generator learns to produce optimal masks that maximize the performance of a downstream model.

Reinforcement Learning Strategy

Directly calculating gradients through the entire training process of a large model is computationally prohibitive. To overcome this, we employ off-policy actor-critic reinforcement learning with entropy regularization. The NMG acts as the "agent," the masking decisions are "actions," and the improvement in task accuracy is the "reward." To ensure training stability, we implement a self-play mechanism where the NMG competes against its previous iterations and random baselines.

Experimental Setup

We evaluated NMG across several diverse benchmarks:

  • Question Answering (QA): SQuAD v1.1 (general), NewsQA (news), and emrQA (medical).
  • Text Classification: IMDb (sentiment) and ChemProt (chemical-protein interactions).

The experiments utilized BERT-base and DistilBERT as the underlying language models to test the framework's effectiveness across different model scales.

Key Results and Analysis

NMG consistently outperformed traditional heuristic masking strategies across nearly all tested datasets and base models.

Performance Gains in Specialized Domains

The benefits of adaptive masking were most pronounced in specialized domains. For instance, on the medical-themed emrQA dataset, NMG achieved significant improvements in both Exact Match (EM) and F1 scores compared to random or rule-based masking. This suggests that NMG successfully identifies and masks complex medical terminology that is crucial for the model to understand the domain.

Comparison of NMG against various masking baselines on SQuAD, emrQA, and NewsQA datasets using BERT and DistilBERT.

Qualitative Insights

Analysis of the generated masks revealed that NMG prioritizes "content" words—specifically nouns and verbs—over functional stop words or punctuation. Interestingly, the optimal strategy was found to be model-dependent; the NMG learned different masking patterns for DistilBERT than for BERT, suggesting that the "difficulty" of the task must be calibrated to the capacity of the model being adapted.

Conclusion

The Neural Mask Generator represents a shift from static, human-designed masking rules toward dynamic, learned adaptation strategies. By leveraging reinforcement learning to identify task-relevant tokens, NMG provides a more efficient and effective path for domain adaptation. Future work will focus on reducing the computational overhead of the RL training process and exploring the potential for zero-shot transfer of learned masking policies across related domains.

Summary updated

Abstract

We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.
BibTeX
@inproceedings{kang2020neural,
  author = {Minki Kang and Moonsu Han and Sung Ju Hwang},
  title = {{Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OLANU_A_BKM},
}

Loading related content...

Figure