For years, the recipe for training bidirectional transformer encoders was treated as settled law: mask 15% of the input tokens and ask the model to predict them. This heuristic, popularized by the original BERT, was based on a delicate balance—masking enough to create a learning signal, but not so much that the model lacked sufficient context to make reasonable guesses.
However, a growing body of research suggests that this 15% threshold is far too conservative. By significantly increasing the masking rate—often to 30%, 40%, or even 80%—researchers are finding that models learn more robust representations in less time.
The Information Density Argument
The primary intuition behind high-density masking is the maximization of the "information signal" per training step. In a standard 15% setup, the model spends a significant amount of its computational budget processing "visible" tokens that it isn't being asked to predict.
Jonas Geiping and Tom Goldstein demonstrated this in their Cramming study, which aimed to train a BERT-style model on a single GPU in 24 hours. They found that raising the masking rate to 25% provided a denser training signal, allowing the model to extract more utility from every processed token. Similarly, Jacob P. Portes and the team behind MosaicBERT adopted a 30% masking ratio, achieving competitive benchmark scores in a fraction of the traditional training time.
Forcing Global Understanding
Beyond mere efficiency, higher masking rates change what the model learns. When only 15% of tokens are missing, a model can often rely on local syntax and immediate neighbors to fill in the blanks. When 50% of the text is gone, the model is forced to develop a more sophisticated, global understanding of the remaining context to reconstruct the missing pieces.
This "difficulty" factor is a recurring theme:
- Robustness: Baohao Liao and colleagues introduced 3ML (Mask More and Mask Later), which uses a 50% masking rate. They argue that this density forces the model to learn more comprehensive representations of the visible context.
- Retrieval: For tasks like dense passage retrieval, Guangyuan Ma showed in CoT-MAE that aggressive masking (30-45%) is more effective because it increases the reconstruction difficulty, leading to deeper semantic embeddings.
- Sentence Embeddings: Chen Xu found in CMLM-CSE that a 40% masking rate in auxiliary tasks makes the global embedding more informative and robust.
Scaling to the Extreme
The most surprising finding in this direction is that the "optimal" masking rate may actually increase as we throw more compute at the problem. In Training compute-optimal transformer encoder models, Megi Dervishi and Yann LeCun challenge the 15% heuristic at scale. Their work suggests that masking ratios as high as 80% can improve downstream performance as the compute budget grows, providing a much more challenging and informative training signal than previously thought possible.
This trend is also reflected in "next-generation" architectures. Lola Le Breton and the NeoBERT team moved to a 20% baseline, noting that the increased difficulty encourages more sophisticated internal representations as training data scales to the trillions of tokens.
Domain-Specific Adaptation
The benefits of high-density masking extend to fine-tuning and domain adaptation. When a model is being adapted to a specialized codebase or a legal dataset, the "local" context is often repetitive or predictable.
Chunqiu Steven Xia explored this in the context of software engineering, finding that 50% masking rates during project-specific fine-tuning prevent the model from over-relying on immediate surrounding text. This forces the model to learn the underlying logic of the specific codebase. Similarly, in the biomedical field, Micheal Abaho demonstrated that exceeding the 15% rate for domain-specific entities provides a more effective signal for learning specialized terminology.
Why It Works: The Disentanglement Intuition
If masking more tokens makes the task harder, why doesn't the model simply fail? The key lies in the realization that the [MASK] token itself can be a bottleneck. In the 3ML approach, Baohao Liao suggests that by "disentangling" the mask—removing the mask tokens from early layers and only introducing them later—the model can focus on processing the visible information first.
By the time the model needs to predict the missing 50%, it has already built a rich representation of the available context. The high masking rate doesn't just make the task harder; it makes the training more efficient by focusing the model's energy on the most informative parts of the sequence.