Low-Resource NER by Data Augmentation With Prompting

Jian Liu, Yufeng Chen, Jinan XuIJCAI 2022 · DOI

TL;DR

This paper introduces a novel data augmentation (DA) technique for Named Entity Recognition (NER) in low-resource settings, leveraging pre-trained language models like BERT. By using "prompting" strategies, they generate more accurate and label-consistent training data, significantly improving NER performance even with very limited labeled examples.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) for identifying and categorizing entities (like people, organizations, locations) within text. However, state-of-the-art NER models typically require large amounts of labeled data for training. This poses a significant challenge in "low-resource" scenarios, where obtaining such extensive labeled datasets is prohibitively expensive or simply impossible, especially in specialized domains. While data augmentation (DA) has shown promise in other NLP tasks, applying it to NER is tricky because token-level accuracy is paramount, and simple word substitutions can easily break the entity-label consistency.

Methods

The core of this paper is a new DA approach for low-resource NER that elicits knowledge from BERT using two prompting mechanisms:

  • Label-Conditioned Word Replacement: This strategy aims to generate more label-consistent examples. It works by masking a word in a sentence and then using BERT to predict a replacement. Crucially, the prediction is conditioned not only on the surrounding context but also on the definition of the entity type (e.g., "Named person or family" for PERSON). This helps ensure that the generated replacement word aligns with the intended entity type.
  • Prompting with Question Answering (QA): This method generates new training data from unlabeled texts by framing NER as a question-answering task. For instance, a prompt like "Who is the named person or family mentioned in texts?" can be used with a BERT model pre-trained on QA datasets to identify PERSON entities. This allows for generating novel examples without relying on existing labeled data.

To combat the noise that can be introduced by these DA methods, the authors also propose an Uncertainty-Guided Self-Training Strategy. This involves iteratively training the NER model, pseudo-annotating the augmented data, and then selecting only the most confident examples to add to the training set. This uncertainty-based selection helps to filter out noisy or incorrect augmentations, leading to a more robust model.

Datasets

The authors evaluate their approach on three standard NER datasets:

  • CoNLL 2003: A widely used dataset with four coarse-grained entity types (PER, LOC, ORG, MISC).
  • OntoNotes 5.0: A more challenging dataset with 18 fine-grained entity types.
  • MaScip: A real-world, low-resource dataset specifically designed for materials synthesis procedures, containing 21 entity types.

They create small (S), medium (M), and large (L) training sets by sampling 50, 150, and 500 sentences, respectively, from these datasets to simulate low-resource conditions.

Key Insights and Ideas

  • Prompting is Powerful for NER DA: The paper demonstrates that carefully crafted prompts can effectively guide pre-trained language models like BERT to generate high-quality, label-consistent data for NER, even in challenging low-resource scenarios.
  • Label Definitions Matter: Conditioning word replacement on entity type definitions (rather than just the label itself) significantly improves the accuracy and relevance of generated examples.
  • QA as a Generator: Framing NER as a QA task is a viable strategy for generating entirely new training instances from unlabeled text, opening up possibilities for zero-shot NER.
  • Uncertainty is Key to Noise Reduction: The uncertainty-guided self-training method is crucial for filtering out noisy augmentations, leading to more stable and effective learning.
  • Complementary Strategies: The two prompting methods (label-conditioned replacement and QA) are complementary, with their combination yielding the best results.

Key Results

The proposed method achieves significant improvements, particularly in low-resource settings:

  • With only 150 training sentences, their approach outperforms previous methods without DA by over 40% in F1 score and prior best DA methods by over 2.0% in F1 score on benchmarks.
  • The method also shows promising results in a zero-shot scenario, where it can achieve decent performance without any human-labeled data for the specific task.
  • The uncertainty-guided self-training strategy helps the model converge faster and achieve higher F1 scores, outperforming simple "easy-first" or "hard-first" selection strategies.

Limitations and Future Work

  • Vague Entity Definitions: The paper notes that entity types with vague or inconsistent definitions (like "MISC" in CoNLL 2003) can degrade performance, as the expressive definitions are less helpful for guiding the augmentation process.
  • Noisy QA Examples: While the QA prompting method can generate good examples, it can also produce flawed ones, highlighting the need for robust noise reduction techniques.
  • Future Directions: The authors suggest applying their method to other related extraction tasks, such as relation extraction, and exploring further improvements to the prompting and uncertainty mechanisms.

Summary updated

BibTeX
@inproceedings{liu2022low,
  author = {Jian Liu and Yufeng Chen and Jinan Xu},
  title = {{Low-Resource NER by Data Augmentation With Prompting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/590},
}
Figure