NerCo: A Contrastive Learning Based Two-Stage Chinese NER Method

Zai Zhang, Bin Shi, Haokun Zhang, Huang Xu, Yaodong Zhang, Yuefei Wu, Bo Dong, Qinghua ZhengIJCAI 2023 · DOI

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, typically framed as a sequence labeling problem using tagging schemes like BIO (Begin, Inside, Outside). In Chinese NER, this task is particularly challenging due to the lack of explicit word delimiters. Standard approaches often suffer from what the authors term Entity Representation Segmentation.

Comparison of standard fine-tuning (left) vs. NerCo's two-stage approach (right), illustrating how contrastive learning prevents entity representation segmentation.

As shown in Figure 1, standard fine-tuning forces the model to treat B- and I- tags as distinct, isolated classes. This leads to fragmented vector representations for tokens belonging to the same entity, scattering them in the embedding space. This "identity crisis" hinders the model's ability to accurately identify entity boundaries and types, especially in noisy or informal text.

The NerCo Framework

To address this, the authors introduce NerCo (Named entity recognition with Contrastive learning), a two-stage training paradigm designed to foster semantic cohesion within entity representations.

Stage 1: Semantic Class Guided Contrast

In the first stage, the model focuses on the broad semantic category of an entity (e.g., PER, ORG, GPE) rather than specific positional BIO tags. Using an InfoNCE loss, the model is trained to pull representations of tokens belonging to the same semantic class closer together while pushing tokens from different classes apart. This ensures that all tokens of a "Person" entity share a cohesive semantic core, regardless of whether they are at the beginning or in the middle of the name.

Stage 2: Token Label Supervised Fine-tuning

Once the encoder has established these cohesive semantic clusters, the model undergoes standard fine-tuning. This stage utilizes the traditional BIO tagging scheme and a Conditional Random Field (CRF) layer to teach the model positional nuances—identifying exactly where an entity starts and ends. By building on the strong semantic foundation from Stage 1, the model can more effectively discriminate between positional labels without losing the underlying entity identity.

Experimental Evaluation

NerCo was evaluated across four major Chinese NER benchmarks:

  • OntoNotes 4.0: A large-scale news-domain corpus.
  • MSRA: A standard news dataset from Microsoft Research Asia.
  • Weibo NER: Noisy social media data containing informal mentions.
  • Resume NER: A specialized dataset of executive resumes.

The framework uses the FLAT Transformer as its base encoder, requiring no additional data augmentation or complex architectural modifications.

Key Results and Insights

NerCo achieved state-of-the-art (SOTA) performance on all tested datasets. The most significant gains were observed on the Weibo NER dataset, where the F1-score improved by 4.11% over the baseline. This suggests that the cohesive representations learned via contrastive learning are particularly beneficial for noisy, small-scale datasets where traditional sequence labeling might struggle.

F1-score performance across different sentence lengths on the Weibo dataset, comparing NerCo with BERT and FLAT baselines.

Furthermore, as illustrated in Figure 6, NerCo demonstrates superior stability when processing long sentences. While traditional models like BERT and FLAT show significant performance drops as sentence length increases, NerCo maintains a higher and more consistent F1-score. This robustness is attributed to the "anchoring" effect of cohesive semantic representations, which helps the model maintain entity context over longer sequences.

Conclusion

NerCo effectively mitigates the problem of entity representation segmentation by separating semantic clustering from positional classification. This two-stage approach leads to more distinct entity clusters and improved boundary detection. Future research may extend this framework to handle nested or discontinuous entities, further broadening its applicability in complex NLP tasks.

Summary updated

BibTeX
@inproceedings{zhang2023nerco,
  author = {Zai Zhang and Bin Shi and Haokun Zhang and Huang Xu and Yaodong Zhang and Yuefei Wu and Bo Dong and Qinghua Zheng},
  title = {{NerCo: A Contrastive Learning Based Two-Stage Chinese NER Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/587},
}
Figure