When training masked language models like BERT, the standard approach is straightforward: select 15% of the tokens uniformly at random and replace them with a [MASK] token. While this simple objective teaches models grammar, syntax, and local word co-occurrences, it often fails to instill a deep understanding of facts, domain-specific concepts, and semantic relationships.
To bridge this gap, a powerful paradigm shift has emerged: Entity-Centric Masking (often referred to as Salient Span Masking). Instead of masking random characters or common stop-words, this strategy targets information-dense phrases—such as named entities, technical terms, dates, and product attributes. By changing what the model is forced to predict, we fundamentally alter how it encodes knowledge within its parameters.
The Problem with Random Shortcuts
To understand why entity-centric masking works, we must first look at the shortcomings of random token masking. Consider the sentence:
"The capital of France is Paris."
If a random masking strategy selects the word "The" or "is", the model can easily fill in the blank using local syntactic rules. Even worse, because modern models use subword tokenization, a random mask might hide only a fragment of a word (e.g., "Pa[MASK]is"). The model can resolve this blank using simple morphological patterns without ever needing to know anything about geography.
As noted by Tang et al. in their work on zero-shot entity linking (A Bidirectional Multi-paragraph Reading Model for Zero-shot Entity Linking), standard random token objectives fail to capture the deeper semantic relationships between entities and their broader contexts. Instead of learning real-world facts, models trained on random tokens frequently rely on superficial word overlaps and local statistical regularities to minimize training loss.
The Mechanism: Targeting Salient Spans
Entity-centric masking replaces randomness with intent. Before masking begins, a document is processed to locate high-value spans. Depending on the implementation, these spans are identified through various means:
- Named Entity Recognition (NER) Tools: Automatically identifying people, locations, or specialized terms (e.g.,
Pergola et al.in Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies). - Dictionary Lookups and Phrase Mining: Extracting domain-specific vocabulary from external databases (e.g.,
Zeng et al.in Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER). - Knowledge Graphs: Sourcing structured entities directly from expert-curated ontologies (e.g., Guanbin Li et al. in Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge).
Once these salient spans are isolated, the entire entity is masked as a single unit.
Standard MLM: The [MASK]pital of France is Pa[MASK]is.
Entity-Aware: The capital of France is [MASK MASK].
Why It Works: Forcing Factual and Conceptual Storage
When an entire entity like "Paris" or a technical term like "Hazard Ratio" is blanked out, the local grammatical shortcuts disappear. The model cannot guess the missing tokens based on adjacent letters or simple sentence structure. Instead, it is forced to perform two higher-level cognitive tasks:
- Contextual Relational Reasoning: The model must analyze the surrounding context ("capital of France") to deduce what kind of entity fits the semantic profile of the sentence.
- Internal Knowledge Retrieval: The model must store the factual connection between the entity and its context directly within its weight parameters to successfully minimize reconstruction error.
In their seminal paper, How Much Knowledge Can You Pack Into the Parameters of a Language Model?, Roberts et al. demonstrated that Salient Span Masking (SSM) drastically improves a model's ability to answer open-domain questions from memory alone, without accessing external text. By focusing the reconstruction task on factual anchors, the model prioritizes world knowledge over generic linguistic patterns. This finding is supported by a comprehensive survey from Youssef et al. (Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models), which highlights that targeting informative phrases is a primary driver for turning language models into reliable "soft knowledge bases."
Even in retrieval-augmented architectures, this mechanism proves vital. Charpentier et al. showed in BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer that masking factual entities provides the necessary training signal to force a model to actually use its external retrieval mechanisms, rather than ignoring them in favor of local context.
Domain Adaptation and Low-Resource Efficiency
The intuition behind entity-centric masking becomes even more compelling in specialized domains like medicine, finance, or e-commerce. In these fields, general vocabulary is secondary to technical concepts. Standard masking wastes precious computational cycles learning common English verbs and stop-words.
In the clinical domain, Miller et al. introduced EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain, which prioritizes masking medical entities like diseases, symptoms, and medications. This forces the model to capture the deep contextual relationships required for complex downstream tasks like temporal relation extraction and negation detection. Similarly, Francis et al. (Clinical entity-aware domain adaptation in low resource setting for inflammatory bowel disease) demonstrated that focusing model attention on specific entities using contextual labels significantly accelerates domain adaptation, especially when data is scarce.
This efficiency makes entity-centric masking a powerful tool for low-resource fine-tuning. Pergola et al. showed that by forcing models to predict missing domain entities during fine-tuning, researchers could achieve massive performance boosts on low-resource question-answering datasets without the prohibitive cost of pre-training a domain-specific model from scratch. The same principle applies to commercial applications: Xu et al. developed K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce to target unique selling propositions and product attributes, ensuring the model's representations capture the most influential aspects of commercial data.
Versatility Across Modalities and Tasks
Because entity-centric masking targets the underlying information structure of data, its benefits extend far beyond standard text paragraphs:
- Structured Data: When training on knowledge graph triples or code, random masking breaks the logical structure.
Moiseev et al.(SKILL: Structured Knowledge Infusion for Large Language Models) applied entity masking to structured triples (subject, relation, object), forcing the model to internalize relational facts. For code and product specifications,Xinze Liet al. developed Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data, masking specific identifiers to compel the model to understand structural dependencies. - Data Augmentation: Entity masking can be inverted or controlled to generate high-quality synthetic data.
Ran Zhou 0004et al. introduced MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER, which selectively masks and predicts entity categories to generate diverse training examples. Conversely,Ghosh et al.(BioAug: Conditional Generation based Data Augmentation for Low-Resource Biomedical NER) preserved the target entities while masking the surrounding context, forcing a generative model to learn the precise linguistic patterns that connect an entity to its environment. - Long-Distance Tracking and Boundaries: In streaming data, masking coreference-related elements like nouns and pronouns encourages models to maintain entity states across long-distance dependencies (
Araujo et al.in A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information). For reading comprehension,Fei Yuanet al. (Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension) demonstrated that masking knowledge phrases sharpens a model's ability to locate exact answer boundaries.
Ultimately, entity-centric masking works because it aligns the training objective with the true goal of language understanding: grasping the core concepts, facts, and relationships that words represent, rather than just the statistical patterns of the words themselves.
Key Foundational Work
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021) — contributed 3 concept elements to this direction.
Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER (2020) — contributed 2 concept elements to this direction.
EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain (2021) — contributed 2 concept elements to this direction.