Exploring the Application of 7B LLMs for Named Entity Recognition in Chinese Ancient Texts

Chenrui Zheng, Yicheng Zhu, Han BiNAACL 2025

TL;DR

This paper explores the application of 7B Large Language Models (LLMs) for Named Entity Recognition (NER) in challenging Chinese ancient texts, particularly in Traditional Chinese Medicine (TCM). By employing domain-specific pre-training and multi-task fine-tuning with LoRA, they achieved a significant F1 score of 0.93, outperforming general-purpose LLMs and establishing a new benchmark for this specialized NLP task.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing entities like people, places, and organizations. While modern LLMs have shown great promise in this area for contemporary languages, applying them to ancient Chinese texts presents unique hurdles. These include archaic vocabulary, flexible grammar, extensive use of homophones, and a scarcity of annotated data. Traditional Chinese Medicine (TCM) texts, in particular, are rife with domain-specific terminology and complex semantics, making them exceptionally difficult for standard NLP models. Existing approaches often struggle to capture the nuances of these historical documents, necessitating specialized methods.

Methods

The researchers propose a comprehensive strategy involving domain-specific pre-training and multi-task fine-tuning. They leverage the LoRA (Low-Rank Adaptation) technique for efficient fine-tuning of LLMs, which significantly reduces computational costs and memory requirements. A key aspect of their approach is domain-specific pre-training using a curated dataset of classical Chinese texts, including historical records and TCM literature. This is complemented by multi-task fine-tuning, where the model is trained on a combined dataset encompassing various tasks (A, B, and C) to enhance generalization capabilities. They also experimented with different prompt engineering strategies, finding that a more structured prompt format (Prompt 2) significantly improved performance.

Datasets

The primary dataset used for this research is derived from the EvaHan2025 competition, specifically focusing on the open modality track. This dataset includes three heterogeneous sources:

  • Shiji ("史记"): Historical records.
  • Twenty-Four Histories ("二十四史"): A collection of historical texts.
  • Traditional Chinese Medicine Classics ("中医药典籍"): A corpus of TCM literature.

For pre-training, they utilized a 12.5MB subset of unannotated classical Chinese text from the open-source project "殆知阁" (Da Zhi Ge), comprising 1/3 historical texts and 2/3 TCM-related texts.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: Pre-training LLMs on relevant historical and domain-specific texts (like TCM literature) significantly boosts performance on specialized NER tasks, far exceeding general-purpose pre-training.
  • Multi-Task Learning Enhances Generalization: Combining data from multiple related tasks (A, B, and C) during fine-tuning helps the model learn more robust representations and generalize better to unseen data within the domain.
  • Smaller Models Can Shine: Surprisingly, a smaller 7B parameter model, when properly trained with domain-specific data and techniques, can achieve competitive or even superior results compared to larger, more general-purpose models like Qwen-Plus on this specialized task.
  • Prompt Engineering Matters: The way prompts are structured can have a substantial impact on LLM performance. A more detailed and contextually rich prompt format led to better results.
  • Ensemble Methods Offer Incremental Gains: Combining predictions from multiple models (7B and 3B) and using a teacher model (Qwen-Plus) for arbitration further improved accuracy, suggesting potential for even greater gains with more advanced reasoning models.

Key Results

The fine-tuned 7B LLM achieved an impressive F1 score of 0.93 on the ancient Chinese text NER task, significantly outperforming general-purpose LLMs. This represents a substantial improvement over baseline methods, such as SikuROBERTa+BiLSTM+CRF, which achieved an F1 score of 0.8333 on Task C (TCM texts). The paper also highlights that Prompt 2 outperformed Prompt 1, with an F1 score of 0.8592 compared to 0.717 for Qwen-Plus. An ensemble of the 7B and 3B models with a teacher model achieved a final F1 score of 0.9277.

Limitations and Future Work

The authors acknowledge several limitations:

  • Teacher Model Choice: Qwen-Plus was used as a teacher model but wasn't extensively compared with other alternatives.
  • Data Annotation Quality: The impact of the quality of data annotations on the results was not discussed.
  • Evaluation Metrics: The evaluation primarily relied on F1 scores, potentially overlooking other important metrics like model robustness and inference speed.
  • Resource Constraints: The study was conducted on a single NVIDIA RTX 4090 GPU, which might limit the scale of experiments.

Future work could involve:

  • Exploring more advanced inference models, such as DeepSeek-R1, to further push performance limits.
  • Investigating the impact of data annotation quality and exploring more comprehensive evaluation metrics.
  • Comparing the chosen teacher model with a wider range of alternatives.
  • Developing more sophisticated ensemble strategies.

Summary updated

BibTeX
@inproceedings{zheng2025exploring,
  author = {Chenrui Zheng and Yicheng Zhu and Han Bi},
  title = {{Exploring the Application of 7B LLMs for Named Entity Recognition in Chinese Ancient Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure