Named Entity Recognition in Context: Edit_Dunhuang team Technical Report for Evahan2025 NER Competition

Colin Brisson, Ayoub Kahfy, Marc Bui, Frédéric ConstantNAACL 2025

TL;DR

The Edit_Dunhuang team developed a novel Named Entity Recognition (NER) system for Classical Chinese texts that leverages external context and a specialized pre-trained language model called Pindola. This approach achieved a significant improvement over the baseline, demonstrating the power of contextual information for historical texts.

What Problem Does This Paper Address?

Named Entity Recognition (NER) in historical texts, particularly Classical Chinese, presents unique challenges. These texts often have specialized vocabularies, complex sentence structures, and a lack of extensive annotated datasets compared to modern languages. Existing NER systems, even those based on powerful transformer architectures, can struggle with the nuances and historical context, leading to inaccuracies. The EvaHan2025 competition specifically aimed to push the boundaries of NER for Classical Chinese across diverse domains, including historical records and medicinal texts.

Methods

The Edit_Dunhuang team's approach is built on three core components:

  1. Pindola: A Specialized Pre-trained Model: They developed "Pindola," a transformer-based bidirectional encoder specifically pre-trained on a massive corpus (3 billion characters) of Classical Chinese texts. This model incorporates several state-of-the-art enhancements for efficient processing of long sequences, including FlashAttention v2, SentencePiece tokenization, SwiGLU activation, and AliBias positional encoding. Two variants were created: Pindola_small and Pindola_large. For the competition, they fine-tuned Pindola_large into Pindola_NER, which includes a token classification head with Bi-LSTM and a CRF layer.
  2. Context Retrieval: Recognizing that target sentences are not isolated, the system retrieves relevant external context. This is achieved by encoding contextual sequences using a specialized variant of Pindola (Pindola_retrieval) and storing them in a vector database. For a given target sentence, similar contextual sequences are identified using L2 distance.
  3. Context Summarization: To make the retrieved context more digestible and useful for the NER model, a generative reasoning step is employed. An OpenAI model is used to generate concise summaries of the retrieved contexts in Classical Chinese. This summarized context is then concatenated with the target sentence as input to the Pindola_NER model.

Datasets

The competition utilized three distinct datasets:

  • Dataset A: Excerpts from the Shiji (historical records), annotated for six entity types: person names, geographical locations, book titles, official titles, country names, and temporal expressions.
  • Dataset B: Excerpts from the Twenty-Four Histories (official dynastic histories), annotated for three entity types: person names, geographical locations, and temporal expressions.
  • Dataset C: Medicinal texts annotated for six specialized entities: disease names, syndromes, medicinal formulas, decoction pieces, symptoms, and acupuncture points.

The team also compiled an additional pretraining dataset by aggregating various publicly available online resources, standardized to match the annotation scheme of Dataset B.

Key Insights and Ideas

  • Context is King (Even for Historical Texts): The paper strongly suggests that integrating external context significantly boosts NER performance, even for historical languages where ambiguity might seem low.
  • Specialized Pre-training Matters: Developing a language model like Pindola, pre-trained specifically on a large corpus of Classical Chinese, is crucial for capturing the nuances of the language.
  • Summarization Bridges the Gap: Using a generative model to summarize retrieved context makes the information more actionable for the NER model, preventing information overload.
  • Domain Specificity Impacts Performance: The model performed best on Dataset B, which aligned well with the pretraining data, and less so on Dataset C (medical texts), highlighting the challenge of underrepresented domains.

Key Results

The Edit_Dunhuang team achieved an overall average F1 score of 85.58, which was nearly 5 points higher than the competition baseline. Their revised submission, after fixing an issue during data preparation, significantly outperformed their initial submission. The ablation study indicated that both external context and pretraining contributed to performance improvements, though the impact was more pronounced on Dataset B.

Limitations and Future Work

The authors acknowledge that due to competition constraints, they couldn't fully optimize every component or conduct exhaustive hyperparameter searches. They also note that the modest benefits from external context might be due to limitations in their retrieval and summarization modules. Future work could explore:

  • Alternative retrieval strategies.
  • Varying approaches to context integration (e.g., minimal or no summarization).
  • Further research into more efficient yet effective NER approaches for low-resource and historical language settings.

Summary updated

BibTeX
@inproceedings{brisson2025named,
  author = {Colin Brisson and Ayoub Kahfy and Marc Bui and Frédéric Constant},
  title = {{Named Entity Recognition in Context: Edit\_Dunhuang team Technical Report for Evahan2025 NER Competition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure