Pretraining Text Encoders with Replaced Token Detection

158 concepts93 papers7 proposals

For years, the dominant recipe for pretraining language models was Masked Language Modeling (MLM), popularized by architectures like BERT. In MLM, a small percentage of tokens (typically 15%) are replaced with a generic [MASK] token, and the model is tasked with reconstructing the original words. While highly effective for learning contextual representations, MLM suffers from a fundamental inefficiency: the model only learns from the 15% of tokens that were masked. The remaining 85% of the sequence acts purely as context, generating no direct error signal or gradient update. Furthermore, the artificial [MASK] token never appears during downstream fine-tuning, creating a distinct mismatch between pretraining and inference.

To resolve these limitations, researchers introduced Replaced Token Detection (RTD). Instead of hiding tokens behind a mask, RTD corrupts the input sequence by replacing some tokens with plausible alternatives. The model is then tasked with a binary classification problem for every single token in the sequence: determining whether it is an original word or a replacement. This shift from generative reconstruction to discriminative verification fundamentally changes both the efficiency of language model pretraining and the structure of the resulting representation space.


The Architecture: Generator-Discriminator Dynamics

The core mechanism of Replaced Token Detection relies on a two-component system: a Generator and a Discriminator.

Original Text:    The  cat   sat  on   the  mat
                    |    |     |    |    |    |
Corruptor (Gen):  The [dog]  sat  on   the [rug]
                    |    |     |    |    |    |
Detector (Disc):   O   R     O    O    O    R
  1. The Generator: Typically a small, lightweight masked language model. It takes an input sequence, masks a subset of tokens, and predicts completions for those masks.
  2. The Corrupted Sequence: The tokens predicted by the generator are slotted back into the sequence. Crucially, if the generator predicts the correct original token, that token is still considered "Original." If it predicts a different but contextually plausible word, it becomes a "Replaced" token.
  3. The Discriminator: This is the primary model being trained for downstream use. It receives the entire corrupted sequence—containing no [MASK] tokens—and predicts a binary label (Original vs. Replaced) for every position.

As detailed by Clark et al. in their seminal paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, this setup resembles a Generative Adversarial Network (GAN) but differs in a crucial way: the generator is trained via maximum likelihood to predict missing tokens, rather than adversarially to fool the discriminator. This avoids the training instabilities typically associated with reinforcement learning or discrete sampling in traditional GANs.


Why It Works: Dense Signals and Plausible Fakes

The primary advantage of RTD is its extraordinary sample efficiency. In a comprehensive overview of contextual representations, Kusner et al. note in A Survey on Contextual Embeddings that discriminative objectives yield a significantly denser training signal per sequence. Because the discriminator computes a loss over 100% of the input tokens rather than just a 15% masked subset, it learns much faster from the same volume of text data.

However, the density of the signal is only half the story; the difficulty of the signal matters equally. If tokens were replaced with entirely random words from the vocabulary, the task would be trivial. For example, in the sentence "The chef cooked a meal," replacing "cooked" with "refrigerator" creates an obvious syntactic and semantic anomaly.

By using a trained language model as the generator, the replacements become highly sophisticated. The generator might replace "cooked" with "ate" or "prepared". As Liu et al. observe in Self-supervised Learning: Generative or Contrastive, forcing a model to distinguish between highly plausible contextual alternatives prevents early training convergence and drives the model to capture deep, fine-grained semantic boundaries. To flag "ate" as a replacement in "The chef ate a meal," the discriminator must understand the subtle real-world pragmatics of what chefs typically do in kitchens.


Reshaping the Representation Space

Beyond raw efficiency, the discriminative objective fundamentally alters how linguistic knowledge is organized within the transformer layers.

In standard masked language models, words that are contextually interchangeable—such as antonyms like "hot" and "cold"—often end up clustered closely together in the vector space because they both satisfy the same [MASK] constraints. RTD breaks this symmetry. Because the discriminator is explicitly penalized if it fails to spot when a contextually plausible but factually or logically incorrect word has been swapped in, it learns to separate these concepts.

Xia et al. demonstrate this phenomenon in Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models, showing that discriminative models are significantly less prone to confusing semantically opposite terms during few-shot prompt completions. Similarly, Shang et al. leverage this property in X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications, finding that RTD-trained models excel at identifying highly nuanced, plausible text clarifications where generative models struggle to differentiate between subtle variations.


Downstream Alignment: The Inductive Bias of Verification

One of the most compelling reasons why RTD works so well across specific downstream applications is task alignment. When we fine-tune a language model to detect errors, evaluate writing, or verify facts, we are asking it to act as a judge. A model pretrained on RTD already possesses an inherent inductive bias for verification.

1. Text Editing and Error Detection

Human proofreading is fundamentally a process of identifying swapped, missing, or incorrect words. Models pretrained to spot artificial token replacements map naturally to these tasks. For instance, Skitalinskaya et al. show in To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support that RTD objectives align tightly with detecting suboptimal phrasing or weak claims in argumentative writing.

This alignment is equally powerful in grammatical error correction pipelines. Both Zheng et al. (An Alignment-Agnostic Model for Chinese Text Error Correction) and Wu et al. (Multi-Class Grammatical Error Detection for Correction: A Tale of Two Systems) demonstrate that using an RTD discriminator as an error detector yields massive performance gains over generative backbones, as the model has spent its entire pretraining phase practicing exactly how to pinpoint linguistic anomalies.

2. Detecting Machine-Generated Text and Hallucinations

As large language models become more prevalent, identifying machine-generated text or factual hallucinations has become critical. Machine-generated text often contains subtle statistical anomalies or contextually jarring token choices that deviate from human writing.

Because an RTD discriminator is trained to spot the "artificial" outputs of a generator model embedded within natural text, it serves as an ideal foundation for these tasks:

3. Code and Specialized Domains

The utility of RTD extends beyond natural language. In programming languages, a single incorrect character or token can completely alter the logic of a program while maintaining syntactic validity. Feng et al. demonstrate in CodeBERT: A Pre-Trained Model for Programming and Natural Languages that combining MLM with RTD forces models to heavily scrutinize code structure, making them highly sensitive to logically flawed but syntactically plausible code mutations.


Conclusion

Replaced Token Detection redefines the efficiency of self-supervised pretraining by transforming a sparse reconstruction task into a dense verification task. By forcing a transformer to evaluate every token against a backdrop of contextually plausible alternatives generated by a secondary network, RTD accelerates convergence and eliminates the pretrain-inference discrepancy. The resulting models do not merely learn what word fits in a blank; they develop a sharp, discriminative understanding of semantic boundaries, making them uniquely robust for downstream tasks centered on verification, error detection, and text quality assessment.

Key Foundational Work

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (2020) — contributed 2 concept elements to this direction.
A Survey on Contextual Embeddings (2020) — contributed 2 concept elements to this direction.
Self-Supervised Learning: Generative or Contrastive (2020) — contributed 4 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 104 papers
  2. Version 1 · 2026-10-04 · 125 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_replaced_token_detection_47bda5e6,
  title = {replaced token detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/replaced-token-detection/rdr_47bda5e6dc15455588774698c1c41dbc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_replaced_token_detection_47bda5e6_v2,
  title = {replaced token detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/replaced-token-detection/rdr_47bda5e6dc15455588774698c1c41dbc/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake