A Prototypical Semantic Decoupling Method via Joint Contrastive Learning for Few-Shot Named Entity Recognition

Guanting Dong, Zechen Wang, Liwen Wang, Daichi Guo, Dayuan Fu, Yuxiang Wu, Chen Zeng, Xuefeng Li, Tingfeng Hui, Keqing He, Xinyue Cui, QiXiang Gao, Weiran Xu

TL;DR

This paper introduces PSDC, a novel method for few-shot Named Entity Recognition (NER) that tackles the problem of models memorizing specific entity mentions rather than understanding context. By decoupling class-specific and contextual semantic information through specialized masking and employing joint contrastive learning, PSDC significantly improves NER performance, especially when labeled data is scarce.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and categorize named entities (like people, organizations, locations) within text. While fully supervised NER models perform well with abundant data, real-world applications often face scenarios with very few labeled examples – the "few-shot" setting. Existing few-shot NER methods often rely on prototypes, where each entity class is represented by an average of its labeled examples. However, these methods suffer from two main issues:

  1. Class Confusion: Prototypes for different classes can become too similar in the model's representation space, leading to misclassifications. This is exacerbated by pre-trained language models (PLMs) that can sometimes collapse representations.
  2. Entity Surface Mention Memorization: Models tend to learn to recognize entities based on their exact surface form (the specific words used) rather than their contextual meaning. This makes them brittle and prone to errors when encountering variations or similar-looking but different entities.

The paper aims to overcome these limitations by enabling models to better leverage limited data for accurate NER.

Methods

The proposed Prototypical Semantic Decoupling method via joint Contrastive learning (PSDC) addresses the identified problems through a two-stage framework:

  1. Span Detection: A standard BERT-based span detector is used to identify potential entity spans within the input text.
  2. Classification Stage (Novelty): This is where PSDC introduces its core innovations:
  3. Semantic Decoupling Masking: Two distinct masking strategies are applied to the input utterance before classification.
  4. Class-specific Masking: All entity spans except those belonging to the target class are replaced with [MASK] tokens. This forces the model to focus on the contextual cues related to the specific entity type being considered.
  5. All Entities Masking: All entity spans are replaced with [MASK] tokens. This creates a representation that captures the broader contextual semantic information of the sentence, independent of specific entity mentions.
  6. Decoupled Prototypes: Based on these masked inputs, two types of prototypes are computed:
  7. Class-specific Prototypes: Averaging representations of entities belonging to the same class.
  8. Contextual Semantic Prototypes: Averaging representations from the "All Entities Masking" inputs, excluding the masked tokens.
  9. Joint Contrastive Learning: To further refine the representation space and ensure effective integration of the decoupled information, PSDC employs two contrastive learning objectives:
  10. Instance-level Contrastive Learning: Pulls representations of similar entities closer and pushes dissimilar ones apart within a batch.
  11. Prototype-level Contrastive Learning: Aligns entity representations with their corresponding class prototypes while pushing them away from other prototypes.

These decoupled prototypes and contrastive losses are then combined to make the final classification decision, aiming to balance class-specific and contextual understanding.

Datasets

The paper evaluates PSDC on two widely used few-shot NER benchmarks:

  • Few-NERD: This dataset features a hierarchy of entity types and includes both "Intra" (entities in train/dev/test belong to different coarse-grained types) and "Inter" (fine-grained types are disjoint across datasets) subtasks.
  • Cross-NER: This benchmark comprises four domains (News, Wiki, Social, Mixed) derived from CoNLL-2003 and other sources. It is used to test the framework's cross-domain generalization capabilities.

No new datasets were introduced in this paper.

Key Insights and Ideas

  • Decoupling is Key: The core idea that separating class-specific cues from broader contextual cues is crucial for robust few-shot NER is a significant insight. This directly addresses the memorization problem by forcing the model to look beyond surface forms.
  • Two Prototypes, Two Perspectives: Using both class-specific and contextual semantic prototypes provides a richer, more nuanced representation for classification. This helps disambiguate entities that might appear similar in isolation but differ in context.
  • Contrastive Learning for Space Optimization: The application of both instance-level and prototype-level contrastive learning is not just for regularization but specifically to combat the "collapsed semantic space" issue, making prototypes more distinct and representations more isotropic.
  • Masking as a Semantic Tool: The clever use of masking strategies to isolate different types of semantic information is a novel and effective way to guide the model's learning process in a low-resource setting.

Key Results

  • State-of-the-Art Performance: PSDC consistently outperforms previous state-of-the-art methods on both Few-NERD (Intra and Inter settings) and Cross-NER benchmarks.
  • Significant Improvements: On Few-NERD, PSDC achieved significant average improvements of 2.26% and 0.68% over MAML-ProtoNet for Intra and Inter settings, respectively.
  • Cross-Domain Robustness: PSDC demonstrated strong performance on Cross-NER, outperforming baselines by 3.89% on average, indicating its ability to generalize across different domains and handle distribution shifts.
  • Ablation Studies: Removing any component of PSDC (class-specific prototypes, contextual prototypes, or contrastive learning objectives) led to a decline in performance, validating the contribution of each part.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the two-stage framework and contrastive learning objectives likely increase computational requirements compared to simpler models.
  • Hyperparameter Sensitivity: The effectiveness of contrastive learning often relies on careful tuning of hyperparameters like temperature and loss weights, which might require extensive experimentation.
  • Future Work: The paper implicitly suggests future work could explore more sophisticated masking strategies or alternative ways to integrate the decoupled semantic information. Further research into optimizing the balance between instance-level and prototype-level contrastive learning could also be beneficial.

Summary updated

BibTeX
@inproceedings{dong2023prototypical,
  author = {Guanting Dong and Zechen Wang and Liwen Wang and Daichi Guo and Dayuan Fu and Yuxiang Wu and Chen Zeng and Xuefeng Li and Tingfeng Hui and Keqing He and Xinyue Cui and QiXiang Gao and Weiran Xu},
  title = {{A Prototypical Semantic Decoupling Method via Joint Contrastive Learning for Few-Shot Named Entity Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095149},
}

Loading related content...

Figure