PMRC: Prompt-Based Machine Reading Comprehension for Few-Shot Named Entity Recognition

Jin Huang, Danfeng Yan, Yuanqiang CaiAAAI 2024 · DOI

Abstract

Few-shot Named Entity Recognition (NER) remains a significant challenge due to the high data requirements of traditional sequence labeling models. We present PMRC (Prompt-based Machine Reading Comprehension), a novel framework that reformulates NER as a single-round reading comprehension task. By leveraging boundary-marker prompts and dot-product similarity, PMRC eliminates the need for fixed classification heads and exhaustive span enumeration. Our approach achieves state-of-the-art performance in low-resource settings, demonstrating superior accuracy and a decoding speed up to 40 times faster than previous template-based methods.

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, involving the identification and categorization of entities such as persons, locations, and organizations. While deep learning has advanced NER significantly, standard models typically require extensive labeled datasets. In "few-shot" scenarios—where only a handful of examples are available—performance often degrades sharply.

Existing prompt-based methods attempt to address this by providing natural language hints to pre-trained language models (PLMs). However, these methods often suffer from high computational complexity due to exhaustive span searching or structural rigidity caused by fixed label heads. PMRC addresses these limitations by introducing a flexible, efficient architecture designed for rapid adaptation to new entity types.

Methodology

PMRC introduces a unified architecture that combines prompt learning with Machine Reading Comprehension (MRC) principles.

The PMRC Architecture

Unlike traditional MRC models that might require multiple queries for different entity types, PMRC uses a single prompt containing boundary markers for all target categories (e.g., [ENT_START] person [ENT_END] [ENT_START] location [ENT_END]).

The PMRC architecture utilizing prompt anchors and dot-product similarity for entity extraction.

The core innovation lies in the use of dot-product similarity. Instead of a standard linear classifier, the model computes the similarity between text token embeddings and the "boundary tokens" within the prompt. These prompt tokens act as learnable prototypes. For instance, if a text token's embedding aligns closely with the [ENT_START] person prototype, it is identified as a potential entity start. This mechanism allows for "zero-parameter adaptation," as new entity types can be introduced simply by updating the prompt text.

Prompting Strategies

The authors explore two primary strategies for constructing these prompts:

  1. Word-based templates: These utilize the semantic names of labels (e.g., "person", "location"), leveraging the latent knowledge within the PLM.
  2. Example-based templates: These incorporate actual labeled snippets from the training set into the prompt to provide richer contextual cues.

To address the inherent class imbalance in NER (where non-entity tokens far outnumber entity tokens), PMRC employs Focal Loss, which prioritizes learning from difficult, rare boundary instances.

Key Results

PMRC was evaluated across diverse benchmarks, including CoNLL03, MIT Movie, MIT Restaurant, and the highly categorical ATIS dataset.

Performance and Efficiency

In cross-domain few-shot experiments, PMRC outperformed existing state-of-the-art (SOTA) models by an average of +5.2%. Its performance was particularly notable on the ATIS dataset, where it achieved a +15.7% improvement in 10-shot settings compared to previous benchmarks.

Beyond accuracy, PMRC offers significant gains in efficiency. It achieves a decoding speed 40.56x faster than TemplateNER and 2.44x faster than LightNER, primarily because it avoids the "n-gram bottleneck" of classifying every possible text span.

Stability Across Entity Scales

A critical advantage of PMRC is its stability as the complexity of the label set increases.

F1-score comparison across varying numbers of entity types, highlighting PMRC's stability.

As shown in the "Class Increment" experiments, PMRC maintains high F1-scores even as the number of entity types grows from 5 to 50. In contrast, competing models like LightNER show performance degradation when faced with a larger variety of categories.

Limitations and Future Work

While PMRC is highly effective, it faces certain constraints. Example-based templates can lead to excessively long input sequences, potentially hitting the maximum token limits of PLMs. Additionally, the model remains somewhat sensitive to the specific choice of "label words" in the prompt. Future research will focus on automating label word selection and extending the high-speed MRC framework to handle nested entities.

Summary updated

BibTeX
@inproceedings{huang2024pmrc,
  author = {Jin Huang and Danfeng Yan and Yuanqiang Cai},
  title = {{PMRC: Prompt-Based Machine Reading Comprehension for Few-Shot Named Entity Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29791},
}
Figure