When training Masked Language Models (MLMs) like BERT, the standard approach is to randomly select individual tokens from the input text, replace them with a special [MASK] token, and task the model with predicting the original identity of those tokens. However, modern language models rarely process text as raw words; instead, they break text down into subword fragments (such as WordPieces) or individual characters.
While subword tokenization solves the problem of out-of-vocabulary words, it introduces a subtle flaw into the self-supervised training objective: the trivial shortcut problem. Whole Word Masking (WWM) is a fundamental modification designed to eliminate this shortcut, forcing models to learn deep semantic and syntactic structures rather than local surface patterns.
The Trivial Shortcut Problem
To understand why standard token-level masking falls short, consider how a word like unbelievable is processed. A subword tokenizer might split this into three distinct tokens: ["un", "##believ", "##able"].
If the model randomly selects tokens to mask, it might mask only the middle fragment: ["un", "[MASK]", "##able"]. To fill in this blank, the model does not need to understand the rest of the sentence, the tone of the paragraph, or the underlying grammar. It simply needs to exploit local morphological collocations—knowing that un- and -able frequently sandwich ##believ. The task degenerates into a local spelling puzzle rather than an exercise in language comprehension.
A similar issue arises in character-based languages like Chinese, where words are composed of multiple characters but tokenized individually. If one character of a multi-character technical term is masked, the model can easily guess it based on intra-word character collocations, bypassing the need to understand the broader context.
This shortcut lowers the difficulty of the pretraining objective. As a result, the model over-indexes on local character or subword correlations, failing to develop robust, long-range contextual representations.
The Mechanism of Whole Word Masking
Whole Word Masking alters the selection strategy during data preparation. Instead of masking tokens independently and at random, the algorithm operates on the level of complete linguistic units. If a single subword token or character is selected for masking, the masking budget is automatically extended to cover all tokens belonging to that same word or semantic unit simultaneously.
Standard Masking: The results were un - [MASK] - able .
Whole Word Masking: The results were [MASK] [MASK] [MASK] .
By erasing the entire word, the local morphological and collocational clues are completely removed. To predict the missing fragments, the model is forced to look outward to the surrounding sentence or document context.
As Martin et al. demonstrate in their development of French language models, this strategy forces the model to learn higher-level lexical representations rather than relying on local subword context. By increasing the intrinsic difficulty of the reconstruction task, the model is driven to build a more sophisticated understanding of how words relate to one another syntactically and semantically.
Cross-Linguistic and Domain-Specific Imperatives
The intuition behind Whole Word Masking becomes even more critical when applied to morphologically rich languages, languages without explicit word boundaries, or highly specialized domains.
Morphologically Rich and Low-Resource Languages
For languages with complex word formations like Basque (Agerri et al.) and Polish (Rybak et al.), subword fragmentation occurs frequently. Applying whole-word masking ensures that complete lexical units are predicted together, creating a more challenging and effective objective for learning robust language representations.
Character-Based Languages Without Word Boundaries
In Chinese NLP, where text is a continuous string of characters, models like MacBERT (Cui et al.) show that masking entire semantic units or multi-character words forces the model to capture higher-level linguistic structures.
This is especially vital for specialized tasks like Chinese proper noun recognition (Ma et al.), agricultural named entity recognition (Tao et al.), and financial sentiment analysis (Liu et al.). In these scenarios, text must first be segmented into words using an external dictionary or tokenizer, and then masked as complete units. This preservation of word integrity allows the model to learn the internal structure and precise boundaries of complex technical terms.
Technical and Specialized Domains
Even within English, specialized vocabularies benefit from this approach. The authors of Difference-Masking: Choosing What to Mask in Continued Pretraining point out that subword masking in technical domains often leaves behind enough linguistic context for the model to guess a missing piece without truly understanding the underlying domain-specific concept. Masking the entire word eliminates this loophole, forcing a deeper reliance on domain context.
Generalizing the Principle Beyond Text Subwords
The core intuition of Whole Word Masking—identifying the true boundaries of a semantic unit and masking it in its entirety to prevent local leakage—extends naturally to other modalities and granularities.
- Morphemes and Phonemes: In low-resource morphological inflection, Wiemerslage et al. demonstrate that forcing a model to predict entire missing morphemes rather than individual characters provides a powerful inductive bias that helps the model capture the underlying rules of word formation. Similarly, in speech synthesis, Li et al. mask entire phoneme sequences corresponding to a single word. This prevents the model from trivially predicting a missing phoneme based on its immediate phonetic neighbors, forcing it to learn broader contextual dependencies.
- Document Images: This concept even transfers to computer vision tasks like document layout analysis. Standard image augmentation techniques like Cutout randomly slice through visual space, creating unnatural artifacts and splitting words in half. To combat this, Lee et al. introduce word-aware masking (DocCutout), which identifies word-level bounding boxes and masks entire semantic units. By preserving the integrity of non-masked words, the model is forced to learn from the surrounding layout and context rather than relying on fragmented character shapes.
Ultimately, by preventing the model from cheating via local surface patterns, Whole Word Masking yields representations that are inherently more robust. For instance, Nookala et al. find that pretraining language encoders with whole-word masking improves their structural modeling, making the downstream models significantly less susceptible to character-level or subword-level adversarial noise.
Key Foundational Work
German’s Next Language Model (2020) — contributed 2 concept elements to this direction.
Give your Text Representation Models some Love: the Case for Basque (2020) — contributed 2 concept elements to this direction.
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020) — contributed 1 concept elements to this direction.