Reformulating Classification as Masked Language Modeling

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2222 papers245 conceptsfrozen 2026-10-04

When we fine-tune a pre-trained language model like BERT for a new task, we typically discard its most valuable asset: the output layer that predicts missing words. In its place, we attach a "classification head"—a randomly initialized linear layer—and force the model to learn a new mapping from scratch.

This traditional approach is inefficient. It treats the model as a feature extractor rather than a knowledgeable agent. A growing body of research suggests a more intuitive alternative: instead of teaching the model to map text to an abstract label like Class 1, we should ask the model to "fill in the blank" using the vocabulary it already knows.

The Cloze-Style Reformulation

The core mechanism of this research direction is the cloze-style prompt. As explored in FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark by Liang Xu 0011 and colleagues, classification is transformed into a completion task.

For example, to classify the sentiment of a movie review, we don't just feed the text into a classifier. We append a template: "The movie was [MASK]." The model then predicts the most likely token to fill that mask. If the model predicts "great," we map that back to a "Positive" label. This process, often called Pattern-Exploiting Training (PET), allows the model to leverage its internal linguistic knowledge to solve the task immediately.

Why It Works: Aligning Objectives

The primary reason this approach succeeds, especially in low-data regimes, is objective alignment. Standard fine-tuning introduces a "cold start" problem: the new classification head has no idea what the model's internal representations mean.

By contrast, prompt-based learning, as described by Muhammad Abdul-Mageed in NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task, aligns the fine-tuning format with the original pre-training objective (Masked Language Modeling). The model is doing exactly what it was trained to do during its billion-word pre-training phase: predicting tokens based on context.

This alignment is a massive "data booster." In How many data points is a prompt worth?, researchers found that a well-crafted prompt can provide a head start equivalent to hundreds or even thousands of manually labeled examples.

Verbalizers and the Mapping Problem

A critical component of this idea is the verbalizer—the bridge between the model's vocabulary and the task's labels. In This joke is [MASK]: Recognizing Humor and Offense with Prompting, Junze Li and Hinrich Schuetze demonstrate that choosing the right "label-indicative tokens" is essential. If you are detecting humor, the mask might map to "funny" or "boring."

This mechanism is flexible enough to handle complex tasks like:

Efficiency in the Real World

The practical impact of this shift is most visible in specialized, low-resource domains. In clinical settings, where data is sensitive and experts are expensive, Phillip Richter-Pechanski demonstrated in Clinical information extraction for lower-resource languages that prompting can boost accuracy from 48% to 84% using only 20 examples.

Similarly, for African languages with limited digital footprints, David Ifeoluwa Adelani showed in MasakhaNEWS that PET can reach 90% of full-supervised performance with just 10 labels per category.

The Lottery of Prompts

While powerful, prompting introduces a new challenge: sensitivity. The exact wording of a template matters immensely. Ning Ding explores this in Exploring Lottery Prompts for Pre-trained Language Models, suggesting that for almost any data point, there exists a "lottery prompt"—a specific sequence of words—that can trigger the correct answer without any parameter updates.

However, as Ananya Kumar and Percy Liang caution in Are Sample-Efficient NLP Models More Robust?, sample efficiency does not always equal robustness. While prompts help models learn from fewer examples, they can still be susceptible to out-of-distribution shifts, reminding us that prompting is a way to access knowledge, not necessarily a way to make that knowledge more resilient.

Papers in this version

Showing 60 of 222 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 227 papers
  2. Version 1 · 2026-10-04 · 222 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cloze_style_task_reformulation_5cd8bbc7,
  title = {cloze-style task reformulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cloze-style-task-reformulation/rdr_5cd8bbc781ca452ab7379c5c56ff9235},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_reformulating_classification_as_masked_l_5cd8bbc7_v1,
  title = {Reformulating Classification as Masked Language Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reformulating-classification-as-masked-language-modeling/rdr_5cd8bbc781ca452ab7379c5c56ff9235/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}