The Pretrain-Finetune Discrepancy of Artificial Tokens

68 concepts55 papers1 proposal

Self-supervised learning has revolutionized machine learning by allowing models to learn rich representations from unlabeled data. The core mechanism behind this success is often an intuitive corruption trick: hide a portion of the input and force the model to reconstruct it. In natural language processing, this manifests as Masked Language Modeling (MLM); in computer vision, it appears as Masked Image Modeling (MIM).

However, this elegant training setup introduces a fundamental paradox known as the pretrain-finetune discrepancy. To hide data, frameworks typically introduce specialized, artificial tokens—such as the ubiquitous [MASK] symbol. Because these artificial tokens never appear in real-world downstream tasks or inference data, the model experiences a severe training-inference distribution shift. The model becomes an expert at processing corrupted sequences, yet it must ultimately deploy on clean, uncorrupted data.

Why Artificial Tokens Degrade Representations

To understand why this discrepancy harms performance, we must look at how a transformer optimizes its internal representations. When a model spends millions of steps processing sequences riddled with [MASK] tokens, its attention mechanisms and parameter allocations adapt to this specific statistical signature.

As Siyuan Li et al. observe in the context of vision architectures, the introduction of learnable mask tokens can inadvertently cause a model to focus its capacity on identifying and processing the mask itself, rather than learning the underlying semantic structure of the image. This creates a representation gap. When the model is transitioned to clean data during fine-tuning, the sudden absence of these structural anchors can cause internal representations to degrade.

Furthermore, this discrepancy represents a fundamental computational inefficiency. In DNA foundation models, Monireh Safari et al. point out that learning to predict tokens that are entirely absent during downstream inference results in an inefficient use of parameters. The model wastes capacity encoding the artificial noise patterns introduced by the training objective rather than focusing purely on the actual biological or linguistic information.

The problem is not limited to text and images. In diffusion language models, Yinjie Wang et al. note that training with random masking patterns creates a severe misalignment with the iterative, block-wise unmasking trajectories preferred during inference. Similarly, Dogyun Park demonstrates that aggressive token dropping—a common strategy to save compute in diffusion transformers—creates a stark representation gap when the model is abruptly transitioned to dense, full-token inputs at inference.

Strategy 1: Eliminating the Mask via Alternative Objectives

The most direct way to solve the pretrain-finetune discrepancy is to abandon artificial mask tokens entirely, ensuring that the model only ever encounters real vocabulary elements during training.

Permutation Language Modeling

Instead of replacing tokens with a [MASK] symbol to achieve bidirectionality, Zhilin Yang et al. designed XLNet. XLNet maximizes the expected likelihood of a sequence across all possible permutations of its factorization order. Because the context is restricted through attention masking rather than token alteration, the model captures bidirectional context while processing standard, uncorrupted text sequences.

Replaced Token Detection

Another highly successful paradigm shift was introduced by Kevin B. Clark et al. with ELECTRA. Instead of masking tokens, the authors use a small generator network to replace select tokens with plausible alternative words. The main model is then tasked with a binary classification problem: determining whether each token is original or replaced. Because the input sequence consists entirely of real words, the training distribution perfectly matches the downstream inference distribution.

Standard MLM:  The [MASK] sat on the mat.  --> (Artificial token shift)
ELECTRA RTD:   The bird sat on the mat.   --> (No artificial tokens)

In the visual domain, Yuxin Fang et al. adapted this philosophy for vision networks by introducing Corrupted Image Modeling (CIM). Rather than using blank mask patches, they replace visual tokens with plausible image corruptions generated by an auxiliary network, successfully eliminating the mask-induced distribution shift for both CNNs and Vision Transformers.

Strategy 2: Smarter Corruptions and Vocabulary Alignment

If a framework must rely on token replacement, researchers have found that substituting masks with actual vocabulary tokens significantly mitigates the distribution shift.

Even the original BERT framework by Jacob Devlin et al. attempted to mitigate this by occasionally replacing chosen tokens with random words or keeping them unchanged. Building on this intuition, Yiming Cui et al. introduced MacBERT for Chinese NLP. Instead of using the [MASK] token, MacBERT replaces targeted words with their synonyms found via a similarity check, ensuring the input text remains structurally natural.

For specialized downstream tasks like translation, Zhen Yang et al. developed Code-Switching Pre-training (CSP). Instead of inserting generic masks into monolingual text, they replace source words with their direct target-language translations. This forces the model to learn cross-lingual alignments while completely avoiding the artificial token bias that degrades downstream translation performance.

Strategy 3: Architectural Isolation

Rather than changing the objective, some approaches physically isolate the artificial tokens from the model's core representation layers.

Traditional Encoder:  [Token 1] [MASK] [Token 3] --> Enters Encoder (Discrepancy)
Isolated Encoder:     [Token 1] ------- [Token 3] --> Enters Encoder (Clean)

In DNA foundation modeling, Monireh Safari et al. implement an asymmetric encoder-decoder design where masked tokens are entirely excluded from the encoder. The encoder only processes the remaining, clean sequence data. Because the encoder never encounters a mask token during pre-training, its learned representations transfer seamlessly to downstream classification tasks without any distribution shift.

A similar architectural boundary is utilized in multi-modal learning. Jihao Liu et al. introduced Unmasked Token Alignment (UTA) for vision-language models. By aligning only the visible, unmasked patches of an image with a frozen teacher model, they bypass the need for mask tokens entirely within the primary visual pipeline, preserving the model's immediate zero-shot capabilities.

Summary

The pretrain-finetune discrepancy highlights a critical lesson in self-supervised learning: the shortcuts we take to construct training signals can become architectural liabilities. Whether by permuting sequence orders, substituting masks with realistic corruptions, or isolating masks from the encoder entirely, bridging this distribution gap ensures that models spend their parameter capacity learning real-world semantics rather than adapting to artificial training artifacts.

Key Foundational Work

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — contributed 2 concept elements to this direction.
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019) — contributed 1 concept elements to this direction.
Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pretrain_finetune_discrepancy_8c1c9b38,
  title = {pretrain-finetune discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pretrain-finetune-discrepancy/rdr_8c1c9b380d944ae5bceb1a68c5ecb13d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_pretrain_finetune_discrepancy_8c1c9b38_v1,
  title = {pretrain-finetune discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pretrain-finetune-discrepancy/rdr_8c1c9b380d944ae5bceb1a68c5ecb13d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake