Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition

Zihan Wang, Kewen Zhao, Zilong Wang, Jingbo Shang

Abstract

Fine-tuning pre-trained language models has recently become a common practice in building NLP models for various tasks, especially few-shot tasks. We argue that under the few-shot setting, formulating fine-tuning closer to the pre-training objectives shall be able to unleash more benefits from the pre-trained language models. In this work, we take few-shot named entity recognition (NER) for a pilot study, where existing fine-tuning strategies are much different from pre-training. We propose a novel few-shot fine-tuning framework for NER, FFF-NER. Specifically, we introduce three new types of tokens, "is-entity", "which-type" and bracket, so we can formulate the NER fine-tuning as (masked) token prediction or generation, depending on the choice of pre-trained language models. In our experiments, we apply FFF-NER to fine-tune both BERT and BART for few-shot NER on several benchmark datasets and observe significant improvements over existing fine-tuning strategies, including sequence labeling, prototype meta-learning, and prompt-based approaches. We further perform a series of ablation studies, showing few-shot NER performance is strongly correlated with the similarity between fine-tuning and pre-training.

Introduction

In the current NLP landscape, fine-tuning pre-trained language models (PLMs) is the standard approach for downstream tasks. However, a significant discrepancy often exists between the objectives used during pre-training and those used during fine-tuning. For example, models like BERT are pre-trained using Masked Language Modeling (MLM)—predicting missing tokens—but are typically fine-tuned for Named Entity Recognition (NER) using a sequence labeling head.

This "objective gap" is particularly problematic in few-shot settings. When only a handful of examples (e.g., 5-shot) are available, the model lacks sufficient data to learn the mapping between its pre-trained linguistic knowledge and the new structural classification task. This study introduces FFF-NER (Formulating Few-shot Fine-tuning for NER), a framework designed to bridge this gap by aligning the fine-tuning process with the model's original pre-training objective.

Methodology: The FFF-NER Framework

FFF-NER reformulates NER from a joint sequence labeling task into a span-based prediction task that mimics pre-training. Instead of processing an entire sentence to identify all entities at once, the model is presented with a "sentence-span pair" and asked to evaluate a specific candidate span.

Specialized Tokens and Task Reformulation

To facilitate this alignment, the framework introduces three specialized token types:

  1. [is-entity]: Used to determine if a specific span constitutes a valid entity.
  2. [which-type]: Used to predict the semantic category of the entity (e.g., Person, Organization).
  3. Brackets []: Used to physically demarcate the candidate span within the sentence, acting as a visual "highlighter" for the model.

Alignment with Pre-training Objectives

The framework is pre-training agnostic, adapting its format to the underlying architecture:

  • Encoder-based (e.g., BERT): The special tokens are treated as [MASK] tokens. The model performs MLM to fill these blanks, leveraging its ability to predict tokens based on context.
  • Encoder-Decoder (e.g., BART): The task is framed as sequence-to-sequence generation. The model generates a version of the sentence that includes the brackets and entity labels, aligning with its original denoising objective.

By decoupling span detection from type prediction, the model handles two simpler, more familiar sub-tasks rather than one complex, joint classification task.

Experimental Setup

The authors evaluated FFF-NER across 10 diverse NER datasets to ensure robustness. These included general news (CoNLL, OntoNotes, WikiGold), social media (WNUT17), and specialized domains such as medical (I2B2), dialogue (Multiwoz), and voice assistants (SNIPS, ATIS). The primary focus was the 5-shot setting, where the model is provided with only five annotated sentences per entity type.

Results and Discussion

Performance Gains

FFF-NER demonstrated substantial improvements over traditional methods. Across all 10 datasets, the framework achieved an average F1 score of 61.15, compared to 50.2 for standard sequence labeling baselines. It also outperformed more complex meta-learning and prompt-based approaches (such as TemplateNER) without requiring manually crafted prompts or external data.

The Alignment Hypothesis

A key finding of the study is that few-shot performance is strongly correlated with the similarity between fine-tuning and pre-training. In ablation studies, when the authors intentionally mismatched the objectives—such as applying the MLM-style FFF-NER to a BART model—performance dropped significantly. This confirms that "alignment" is a primary driver of success in low-resource scenarios.

The Role of Brackets and Negative Sampling

Structural elements proved critical. Removing the brackets around candidate spans led to a decrease in performance, suggesting that these markers help the model focus its attention without disrupting the natural flow of the sentence. Additionally, the authors found that a weighted negative sampling strategy—prioritizing "hard negatives" (spans that partially overlap with real entities)—was essential for teaching the model precision.

Conclusion and Future Work

FFF-NER demonstrates that the key to unlocking the latent knowledge within PLMs for few-shot tasks lies in objective alignment. By making the fine-tuning task look like pre-training, models can generalize more effectively from minimal data.

While FFF-NER is computationally more intensive during inference due to its span-based nature (evaluating multiple candidate spans per sentence), the performance gains in data-scarce environments are significant. Future work will explore applying this alignment principle to other NLP tasks and investigating how to incorporate semantic label descriptions to further enhance few-shot capabilities.

Summary updated

Abstract

Fine-tuning pre-trained language models has recently become a common practice in building NLP models for various tasks, especially few-shot tasks. We argue that under the few-shot setting, formulating fine-tuning closer to the pre-training objectives shall be able to unleash more benefits from the pre-trained language models. In this work, we take few-shot named entity recognition (NER) for a pilot study, where existing fine-tuning strategies are much different from pre-training. We propose a novel few-shot fine-tuning framework for NER, FFF-NER. Specifically, we introduce three new types of tokens, "is-entity", "which-type" and bracket, so we can formulate the NER fine-tuning as (masked) token prediction or generation, depending on the choice of pre-trained language models. In our experiments, we apply FFF-NER to fine-tune both BERT and BART for few-shot NER on several benchmark datasets and observe significant improvements over existing fine-tuning strategies, including sequence labeling, prototype meta-learning, and prompt-based approaches. We further perform a series of ablation studies, showing few-shot NER performance is strongly correlated with the similarity between fine-tuning and pre-training.
BibTeX
@inproceedings{wang2022formulating,
  author = {Zihan Wang and Kewen Zhao and Zilong Wang and Jingbo Shang},
  title = {{Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.232},
}
Figure