Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition

Zihan Wang, Kewen Zhao, Zilong Wang, Jingbo Shang

Introduction

In Natural Language Processing (NLP), the standard paradigm involves pre-training large-scale models on vast corpora followed by fine-tuning for specific downstream tasks like Named Entity Recognition (NER). While effective with abundant data, this process often falters in "few-shot" scenarios where only a handful of examples are available. Researchers from UC San Diego argue that this struggle stems from a fundamental "objective gap" between the pre-training task (e.g., predicting masked tokens) and the fine-tuning task (e.g., sequence labeling).

The FFF-NER Framework

To address this gap, the authors propose FFF-NER (Formulating Few-shot Fine-tuning for NER). The core philosophy is to reformulate the NER task to mirror the model's native pre-training logic. Instead of treating NER as a sequence labeling problem—a format the model never encountered during pre-training—FFF-NER casts it as a "fill-in-the-blank" or "text-generation" task.

Overview of the FFF-NER framework for BERT-like and BART-like models compared to standard methods.

Implementation for BERT and BART

For BERT-like models (Masked Language Modeling), FFF-NER wraps potential entity spans in brackets and appends special tokens: [is-entity] and [which-type]. The model uses its pre-trained mask prediction head to determine if a span is an entity and its corresponding category. For BART-like models (Sequence-to-Sequence), the task becomes a denoising problem where the model generates a "clean" version of a sentence containing entity labels within brackets (e.g., [ Tom | Person ]).

Key innovations include decoupling entity detection from classification, using brackets as structural metadata, and employing strategic negative sampling to handle non-entity spans without overwhelming the model.

Experimental Results

The framework was evaluated across 10 diverse NER datasets, including news (CoNLL, OntoNotes) and domain-specific sets (MIT Movie, I2B2), primarily in a 5-shot setting.

Performance comparison of FFF-NER against various baselines across multiple datasets and language models.

As shown in the experimental results, FFF-NER consistently outperforms standard sequence labeling and other few-shot methods. Across all 10 datasets, FFF-NER achieved an average F1 score of 61.15, a significant improvement over the 50.2 achieved by traditional sequence labeling. Notable gains were observed in challenging datasets like Multiwoz, where performance surged from 12.3 to 47.56 F1.

Analysis and Insights

The study highlights that task alignment is crucial for few-shot success. Performance gains were strongly correlated with how closely the fine-tuning objective matched the pre-training one. Ablation studies further revealed that structural cues like brackets are vital, acting as signals for the model to focus on specific task-related information. By leveraging the model's existing "heads" developed during pre-training, FFF-NER avoids the need to learn complex new layers from scratch with minimal data.

Conclusion and Future Work

FFF-NER demonstrates that "speaking the model's language" is a powerful strategy for few-shot learning. While the current implementation for BERT-like models faces some inference speed challenges due to span-checking, the underlying "alignment principle" offers a promising path forward. Future research may explore using natural language labels (e.g., the word "Person") instead of special tokens and extending this approach to other complex NLP tasks like relation extraction.

Summary updated

Abstract

Fine-tuning pre-trained language models has recently become a common practice in building NLP models for various tasks, especially few-shot tasks. We argue that under the few-shot setting, formulating fine-tuning closer to the pre-training objectives shall be able to unleash more benefits from the pre-trained language models. In this work, we take few-shot named entity recognition (NER) for a pilot study, where existing fine-tuning strategies are much different from pre-training. We propose a novel few-shot fine-tuning framework for NER, FFF-NER. Specifically, we introduce three new types of tokens, "is-entity", "which-type" and bracket, so we can formulate the NER fine-tuning as (masked) token prediction or generation, depending on the choice of pre-trained language models. In our experiments, we apply FFF-NER to fine-tune both BERT and BART for few-shot NER on several benchmark datasets and observe significant improvements over existing fine-tuning strategies, including sequence labeling, prototype meta-learning, and prompt-based approaches. We further perform a series of ablation studies, showing few-shot NER performance is strongly correlated with the similarity between fine-tuning and pre-training.
BibTeX
@inproceedings{wang2022formulating,
  author = {Zihan Wang and Kewen Zhao and Zilong Wang and Jingbo Shang},
  title = {{Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.232},
}
Figure