Masking What Gives the Domain Away

When a text classifier trained on one kind of text, say book reviews, is applied to another, say airline reviews, it often fails. The target domain carries words the source never taught it to read ("legroom", "delayed", "turbulence"). A common fix is to keep pre-training the language model on unlabeled text from both domains with masked language modeling (MLM). MLM hides some words and trains the model to guess them from context. Standard MLM hides words uniformly at random, so most of the training signal goes to words like "the" and "of" that the model already knows. This proposal chooses which words to hide using a small probe, the domain discriminator, which reads the model's internal representation of each word and predicts which domain the text came from. The words it finds most revealing are masked more often. As the model learns to rebuild those words from context shared across domains, their representations become less domain-specific. The probe is then retrained and moves on to whatever still gives the domain away.

The opportunity comes from an unresolved disagreement in prior work. One line masks words that are shared across domains and predictive of the label: the "pivots" of PERL and PBLM. Another masks words that distinguish the new domain, as in Difference-Masking. Nobody has compared the two under one budget. The adversarial domain adaptation literature, meanwhile, uses the domain discriminator only as a gradient penalty. It never uses the discriminator to decide what to mask. Our approach needs no target labels. Because the probe predicts domain rather than task, it also needs no task labels to choose tokens, so one adapted encoder can serve any downstream task.

Core Claim

A domain discriminator that operates on individual words works better as a masking policy than as a gradient penalty. Masking the words it finds most domain-revealing, in both source and target text, and refreshing that choice as training proceeds should give better cross-domain transfer than three alternatives. The first is random masking. The second is PERL-style pivot masking, which needs source labels to pick tokens. The third is a static, count-based list of domain-distinctive words. We also expect this to be more stable than standard adversarial alignment, because no gradient is ever reversed. The encoder and the probe each minimize ordinary losses. The only "adversarial" element is in data selection: the probe keeps finding what remains domain-specific, and masking forces the encoder to rebuild it from context.

Mechanism

Continued pre-training runs on pooled unlabeled source and target text. Every $K$ steps we freeze a snapshot of the encoder and fit a linear probe on its contextual word vectors from held-out, unmasked sentences, predicting source versus target. For position $i$ with probe log-odds $\ell_i$, the masking probability is

$$p_i = (1-\lambda)\,\rho + \lambda \cdot \min\!\Big(1,\ \rho\, n\, \frac{\exp(|\ell_i|/\tau)}{\sum_{j=1}^{n}\exp(|\ell_j|/\tau)}\Big).$$

Here $n$ is the sequence length, $\rho$ is the average masking budget (15% by default), $\tau$ is a temperature controlling how sharply mass concentrates on revealing words, and $\lambda$ is the share of the budget given to probe-guided masking rather than uniform masking.

The uniform share is there on purpose. The file flags over-specialization and loss of general fluency as a known risk of keyword-only masking. Masking is done over whole words to avoid the sub-word shortcut noted in Direction 1. After continued pre-training, a classifier is fine-tuned on labeled source data only and evaluated on the target, following PERL's protocol.

Relation to Prior Work

Several papers motivate the premise that masking choices matter. NB-MLM weights masking by Naive Bayes word importance and reports a 3.85% error rate on IMDB versus 4.19% for uniform masking. Train No Evil trains a neural selector from drops in classifier confidence. Do not Mask Randomly masks KeyBERT keywords at 75%. Choose Your Words Wisely masks ConceptNet concepts and cuts compute sharply against BioBERT. All four adapt to a single domain, and the first two need labels to score tokens. Our probe needs neither labels nor an external lexicon. It targets cross-domain transfer rather than in-domain efficiency.

Difference-Masking is the closest token-selection method. It scores words by how much more frequent they are in the target corpus than in a web-scale background corpus, then spreads masking to embedding neighbors. Our method differs in two ways. The contrast is between source and target, not target and the general web. The scores come from a probe on contextual representations and are updated during training, rather than from fixed surface counts. The file also notes that Difference-Masking "has no mechanism to establish invariance." We test whether contrast-driven masking of both domains supplies one.

PERL and PBLM supply the competing hypothesis and the evaluation protocol. They mask shared, label-predictive pivots, and pivot selection uses source labels. We mask the opposite set, the domain-private words, and ask which choice transfers better.

Adversarial adaptation supplies the discriminator, not the training rule. Domain-adversarial training with a gradient reversal layer ( Adversarial Discriminative Domain Adaptation, Conditional Adversarial Domain Adaptation; Conditional Adversarial Domain Adaptation), and text variants that align whole-sentence vectors ( Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring), are known to be unstable and to align the wrong classes. Those failures constrain our design and are why we avoid reversed gradients. ADIOS shows that a learned masker can force a vision encoder to model semantic parts. It uses a true min-max game, works on images, and does not transfer across domains. We keep the idea of a learned mask chooser but replace the game with a periodically refit probe.

On the target-data axis, test-time methods such as EATA and Mask-Defense avoid pre-collected target data entirely. Our method still needs an unlabeled target corpus, so we do not claim target-free adaptation. Instead we measure how little unlabeled target text the probe needs, which is the practical form of the dependency.

Research Program

The work has three parts. First, we build the probe-guided masking loop on BERT-base and reproduce PERL's cross-domain sentiment setup, so every comparison shares PERL's unlabeled data and label splits.

Second, we run a controlled study of what to mask at a fixed 15% budget and fixed number of pre-training steps. The five conditions are uniform random tokens, PERL pivots, static source-versus-target count contrast, probe-guided with a single refresh, and probe-guided with periodic refresh. This directly settles the pivot-versus-difference tension in the file.

Third, we analyze the mechanism. We track probe accuracy on the final encoder's representations, which should fall if masking produces invariance. We track which word types the probe selects over time. We also measure accuracy against unlabeled target corpus size at 100, 1,000, and the full PERL target set.

A secondary in-domain check sets the "source" to a general corpus and the "target" to a specialist domain. In that setting the method becomes a learned version of Difference-Masking, and we test on ACL-ARC and PUBHEALTH to confirm it does not degrade single-domain adaptation.

Baselines

The naive reference point is BERT fine-tuned on labeled source reviews and applied directly to the target. PERL reports this at 84.1% average accuracy on the Books/DVD/Electronics/Kitchen pairs. A second reference is continued pre-training on pooled source and target text with uniform 15% masking, using the same steps and data as our method. The file contains no reported result for this on the product-review pairs, so we run it ourselves. On PUBHEALTH, Do not Mask Randomly reports 64.94% F1 for random-masking pre-training.

The strongest existing competitor is R-PERL, which reports 87.5% average accuracy on the same product-review pairs and about 84% on the product-to-airline shift. It is the strongest because it is the best reported masking-based method on exactly this benchmark, and it uses more supervision than we do (source labels for pivot selection). We rerun it from its protocol with the same encoder, unlabeled data, pre-training steps, five seeds, and an equal hyperparameter search budget. Beating it means matching or exceeding 87.5% on average without using labels to pick tokens, and clearly exceeding it on the airline shift, where domain-private vocabulary matters most.

Domain-adversarial BERT is the other main competitor, with a gradient reversal layer on the sentence vector during source fine-tuning plus pooled unlabeled target text. It is the canonical use of a discriminator, and the file contains no reported result for it on these review pairs, so we run it with the same tuning budget. We report the across-seed variance for both methods, because stability is part of our claim.

For in-domain transfer, Difference-Masking's reported 74.04% accuracy on ACL-ARC, against 63.74% for random masking, is the number to approach.

The key ablation replaces the refreshed probe with a static source-versus-target count list at the same budget. It isolates whether dynamic, representation-level selection adds anything beyond masking domain-distinctive words. We also run a variant that adds gradient reversal on the token probe, to test whether an explicit penalty adds to masking.

Why not just…?

Why not just use PERL? PERL is a real threat. If probe-guided masking only matches R-PERL's 87.5%, the case rests on needing no labels for token selection and on producing a task-agnostic encoder. That is a narrower but still useful result. The decisive test is the head-to-head at equal budget, especially on the airline shift. Pivots are by definition shared words, so they may miss exactly the target-private vocabulary that causes the shift.

Why not just add gradient reversal to BERT? The file documents that sentence-level adversarial alignment can match non-corresponding classes and is sensitive to the balance between discriminator and encoder learning rates. Our experiments compare mean accuracy and seed variance directly against domain-adversarial BERT. The gradient-reversal variant of our own method shows whether the penalty helps once masking is already in place.

Why not just count words that differ between source and target and mask those? This is the most serious simple alternative: essentially Difference-Masking with the source as the background corpus. If it performs within noise of the refreshed probe, we will say so. The contribution then becomes the empirical resolution that masking domain-private words beats masking pivots, and the learned probe is unnecessary. The static-list ablation is designed to show this outcome if it holds.

Why not just drop domain-revealing words during source fine-tuning? Deleting source-private cues could stop the classifier from relying on them, with no extra pre-training at all. We run this as a cheap ablation, using the same probe scores to drop words at fine-tuning time. If it recovers most of the gain, the benefit comes from removing features rather than from learning representations.

Decisive Checks

The idea holds if three things are true. Probe-guided masking beats uniform masking and PERL pivots at identical budget on the product-review pairs and the airline shift, averaged over five seeds. The refreshed probe beats the static count list. Domain-probe accuracy on the final representations drops, while in-domain results on ACL-ARC and PUBHEALTH stay at or above random-masking levels.

The idea is falsified in two cases. If pivots win, the "shared words" side of the tension is right. If static counts tie with the probe, the dynamic, discriminator-driven part adds nothing. In that case the paper reduces to the finding that one should mask domain-private words rather than shared ones.