Difference-Masking: Choosing What to Mask in Continued Pretraining

Alex Wilf, Syeda Nahida Akter, Leena Mathur, Paul Pu Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency

TL;DR

Difference-Masking is a novel self-supervised strategy that improves continued pretraining by automatically selecting what to mask based on what makes a target domain unique. By identifying domain-specific "anchors" and prioritizing them for masking, this approach consistently outperforms random masking and sophisticated baselines across both language and multimodal video tasks.

What Problem Does This Paper Address?

Self-supervised learning (SSL) via "mask-and-predict" (like BERT) has revolutionized AI. However, most models still rely on random masking, where any token has an equal chance of being hidden. The authors argue this is inefficient, especially during continued pretraining—the phase where a general-purpose model is adapted to a specific domain (like chemistry or social intelligence).

Think of a chemistry student taking a fill-in-the-blank test. A teacher wouldn't mask common words like "the" or "process"; they would mask domain-specific terms like "catalyst" or "polymer" to truly test and build understanding. Current SSL lacks this intuition. While some methods use specialized tools like Named Entity Recognizers (NER) to guide masking, these are expensive to build and don't exist for every domain. The challenge is to automatically determine what is "important" to mask in a new domain without using human-labeled data.

Methods

The authors introduce Difference-Masking, a two-step, entirely self-supervised process:

  1. Finding Difference Anchors (TF-ICF): The system identifies words that are frequent in the target domain ($X_T$) but rare in general language ($X_{PT}$). They use a metric called TF-ICF (Term-Frequency, Inverse-Corpus-Frequency), a variation of the classic TF-IDF. For example, in a chemistry corpus, "molecule" would have a high TF-ICF score compared to a general web corpus. The top $K$ words become "anchors."
  2. Similarity-Based Masking: During training, the model doesn't just mask the anchors. Instead, it calculates the similarity between every word in a sentence and these anchors using BERT embeddings. Words that are semantically close to the anchors are assigned a higher probability of being masked.
Figure 1: DIFFERENCE-MASKING automatically se- lects what to mask based on what makes the task domain different from the pretraining domain, enhancing model learning on the end task.

For Multimodal Video, the authors apply a visual analogue. Instead of masking random 16x16 image patches (which might just be background noise), they mask semantic objects within bounding boxes. In social intelligence tasks, for instance, the "anchors" are often people, so the model prioritizes masking faces and bodies.

The Difference-Masking workflow for language and video

Datasets

The authors evaluated their method on four diverse datasets:

  • ACL-ARC: A citation intent task using scientific papers from the ACL Anthology.
  • ChemProt: A low-resource chemical-protein interaction relation classification task.
  • TVQA: A large-scale video QA dataset based on 6 American TV shows.
  • Social-IQ: A challenging benchmark for artificial social intelligence, requiring reasoning about social interactions in videos.

Key Insights and Ideas

  • The Uniqueness Hypothesis: The core idea is that concepts unique to a domain ($X_{T/PT}$) share higher mutual information with the eventual task labels than concepts common to all domains. Masking these "unique" parts forces the model to learn the most relevant representations.
  • Recovering Task Labels Automatically: In the ChemProt dataset, the algorithm naturally chose to mask words like "inhibitor," "activator," and "antagonist." Remarkably, these are the actual labels for the downstream task. The model "discovered" what was important about the task without ever seeing a label.
  • Semantic vs. Random Visual Masking: In video tasks, masking based on semantic entities (like people in social scenes) proved far more effective than masking random pixels or using general-purpose vision-language models like CLIP to guide masking.
Frequently masked words vs. actual task labels/tracks

Key Results

Difference-Masking outperformed all baselines, including random masking and methods that use supervised labels (like Selective Masking):

  • ACL-ARC: Achieved 74.04% accuracy, a massive jump from the 63.74% random baseline and significantly better than the best previous baseline (71.94%).
  • ChemProt: Reached 83.94%, outperforming both random masking and EntityBERT.
  • Social-IQ: Achieved 71.37%, notably outperforming AttnMask (70.18%) and random masking (69.05%).
  • TVQA: Reached 81.73%, showing a substantial gain over the 73.75% random baseline.

The authors also found a "sweet spot" for the number of anchors ($K=20$) that worked across different datasets, suggesting a level of robustness in the hyperparameter.

Sensitivity analysis of the number of anchors (K)

Limitations and Future Work

While effective, the authors acknowledge that TF-ICF is a simple approximation of what makes a domain unique. Future work could explore more sophisticated ways to define the "difference" between datasets.

Additionally, while the method works well for multimodal data, it currently relies on object detectors to define bounding boxes. Developing a version that identifies "visual anchors" in a completely end-to-end, unsupervised way remains an open challenge. Finally, the authors noted that the performance margin varied between tasks (e.g., a larger gain on Social-IQ than TVQA), suggesting that a deeper investigation into why certain tasks benefit more from difference-masking is needed.

Summary updated

Abstract

The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain on domain-specific data before performing some downstream task. We introduce Difference-Masking, a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. Empirically, we find that Difference-Masking outperforms baselines on continued pretraining settings across four diverse language-only and multimodal video tasks.
BibTeX
@inproceedings{wilf2023difference,
  author = {Alex Wilf and Syeda Nahida Akter and Leena Mathur and Paul Pu Liang and Sheryl Mathew and Mengrou Shou and Eric Nyberg and Louis-Philippe Morency},
  title = {{Difference-Masking: Choosing What to Mask in Continued Pretraining}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.881},
}
Figure