Classification as Masked Language Modeling

275 concepts205 papers5 proposals

When we fine-tune a pre-trained language model for a downstream classification task, we traditionally perform a piece of surgery: we chop off the model's original output layer (the masked language modeling head) and replace it with a randomly initialized linear classification head.

This approach introduces a fundamental objective gap. The model spends its entire pre-training life learning to predict missing words in a context. When we suddenly force it to map an entire sequence to an arbitrary class label via a newly added layer, it must learn a completely new mechanism from scratch. In low-resource or few-shot scenarios, this objective gap makes fine-tuning highly inefficient, as the model struggles to align its rich latent knowledge with the new classification head.

To solve this, a powerful paradigm reformulates classification tasks into cloze-style (fill-in-the-blank) tasks. Instead of changing the model to fit the task, we change the task to fit the model.


The Anatomy of Cloze Reformulation

To turn a classification task into a language modeling task, we wrap the input text in a natural language template (a prompt) that contains a [MASK] token. We then define a mapping (a verbalizer) that links specific words in the model's vocabulary to our target classes.

Consider a sentiment analysis task evaluating movie reviews. Instead of passing the text through a linear layer to output a 0 or 1, we reformulate the input:

$$\text{Input: "The cinematography was breathtaking."} \longrightarrow \text{Template: "The cinematography was breathtaking. It was [MASK]. "}$$

We then look at the model's predicted probability distribution over its entire vocabulary for that [MASK] token. If the model predicts the token "great", we map it to the Positive class; if it predicts "terrible", we map it to the Negative class.

By utilizing this pattern-verbalizer framework, as explored by Derek Tam et al. in Improving and Simplifying Pattern Exploiting Training, we completely bypass the need for a new classification head. The downstream task becomes identical in format to the original pre-training objective.


Why It Works: Maximizing Latent Knowledge

The primary reason cloze-style reformulation works so well—particularly in data-scarce environments—is that it preserves and exploits the model's latent semantic knowledge.

When a model is pre-trained on billions of sentences, it implicitly learns complex real-world relationships, syntax, and subtle linguistic nuances. As Jiaying Wu et al. note in Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection, traditional fine-tuning can distort or overwrite these pre-trained weights because the gradients from a randomly initialized head propagate backwards with significant disruption.

By contrast, cloze prompting queries the model using the exact same linguistic patterns it has already mastered. As argued by Yanmeng Wang et al. in PINGAN Omini-Sinitic at SemEval-2022 Task 4: Multi-prompt Training for Patronizing and Condescending Language Detection, this alignment is uniquely effective for identifying subtle, implicit language structures (such as condescension or humor) that a standard classification head might overlook. In few-shot scenarios with only a handful of examples, treating prediction as a natural text-completion exercise prevents overfitting and yields significantly higher accuracy.


Beyond Simple Classification: Relational and Structural Tasks

While sentiment analysis provides an easy mental model, this paradigm extends naturally to highly complex, relational NLP tasks by altering where the mask is placed and how the verbalizer behaves.

Implicit Discourse Relations

Identifying the logical connection between two sentences (e.g., comparison, cause-and-effect) is notoriously difficult when explicit transition words like but or because are missing. Hao Zhou et al. demonstrate in Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition that this can be solved by inserting a mask token directly between the two text segments:

$$\text{"I stayed inside. [MASK] it was raining."}$$

If the model predicts "because", the system maps this token to a Causal discourse relation. This allows the model to leverage its pre-trained understanding of sentence transitions to perform implicit classification.

Coreference and Relation Extraction

We can also use cloze prompts to determine if two entities in a text interact or refer to the same event. In CorefPrompt: Prompt-based Event Coreference Resolution by Measuring Event Type and Argument Compatibilities, Sheng Xu et al. show that inserting prompt templates requiring a model to predict identity or compatibility via mask tokens allows feature extraction and relational judgment to happen simultaneously within a single contextual window. Similarly, Jiaxin Wang et al. in MatchPrompt: Prompt-based Open Relation Extraction with Semantic Consistency Guided Clustering isolate relational semantics by placing a mask token between two entities, using the resulting hidden state to cluster and discover open-ended relationships.


Optimizing the Verbalizer

One engineering challenge in this paradigm is selecting the right words for the verbalizer. If you map the Positive class to "excellent" but the model prefers the word "good", your classification accuracy will suffer simply due to vocabulary mismatch.

While early work relied on manual engineering, modern implementations often automate this process. As discussed by Robert K. Helmeczi et al. in Few shot learning approaches to essay scoring, discrete search algorithms can scan a model's vocabulary to automatically find the optimal verbalizer tokens—maximizing the mathematical alignment between the model's internal representations and the target labels.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 227 papers
  2. Version 1 · 2026-10-04 · 222 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cloze_style_task_reformulation_5cd8bbc7,
  title = {cloze-style task reformulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cloze-style-task-reformulation/rdr_5cd8bbc781ca452ab7379c5c56ff9235},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_cloze_style_task_reformulation_5cd8bbc7_v2,
  title = {cloze-style task reformulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cloze-style-task-reformulation/rdr_5cd8bbc781ca452ab7379c5c56ff9235/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake