Label-Guided In-Context Learning for Named Entity Recognition

Fan Bai, Hamid Hassanzadeh, Ardavan Saeedi, Mark DredzearXiv 2025 · DOI · arXiv

Abstract

Named Entity Recognition (NER) is a critical task for identifying specific tokens within text, yet Large Language Models (LLMs) often struggle to match the performance of specialized, fine-tuned models when using standard In-Context Learning (ICL). This paper introduces DEER (Data stat.-grounded namE Entity Recognition), a training-free framework that leverages label-grounded statistics to guide both demonstration retrieval and a targeted error-reflection process. By prioritizing relevant entity and context patterns, DEER significantly boosts NER performance across diverse benchmarks, achieving results comparable to supervised fine-tuning without requiring model updates.

Introduction

Named Entity Recognition (NER) requires precise, token-level identification of entities such as names, locations, or medical terms. While LLMs excel at general language tasks, their performance on NER via ICL is often limited by suboptimal demonstration retrieval. Most existing methods rely on task-agnostic semantic similarity, selecting examples that sound like the test query but may contain entirely different entity patterns. Furthermore, standard ICL typically follows a single-pass inference process, leaving no room for the model to correct errors on ambiguous or rare tokens.

Comparison between standard ICL-based NER and the proposed DEER framework.

To address these bottlenecks, we propose DEER, which utilizes training set labels as a signal rather than just a target. By analyzing the distribution of labels across tokens, DEER identifies the most relevant demonstrations and implements a reflection step to refine initial predictions.

Methodology

DEER is a three-stage, training-free framework designed to maximize the utility of available training data during inference.

1. Data Statistics Preparation

Before inference, the framework analyzes the training set to compute frequencies and probabilities for three token categories: entity tokens (part of a label), context tokens (words immediately surrounding an entity), and other tokens. This statistical profile serves as the foundation for the subsequent retrieval and reflection steps.

2. Label-Guided Demonstration Retrieval

Unlike traditional methods that use sentence-level embeddings, DEER uses pre-computed statistics to prioritize training examples that share relevant entity or context patterns with the test query. This ensures the LLM is provided with demonstrations that are contextually and structurally relevant to the specific entities it must identify.

3. Error Reflection

Following an initial prediction, DEER identifies "error-prone" tokens—such as unseen words in entity-rich contexts or tokens frequently labeled as entities in training but missed in the current pass. The model is then prompted to reflect on these specific tokens using targeted, span-level demonstrations to produce an updated, more accurate prediction.

The DEER workflow: from label-guided retrieval to initial prediction and error reflection.

Experimental Results

DEER was evaluated across five diverse datasets (CoNLL03, OntoNotes, NCBI-disease, BC2GM, and TweetNER7) using state-of-the-art LLMs including Qwen2.5, Llama3.3, and GPT-4o.

  • Performance Gains: DEER consistently outperformed standard ICL baselines. For example, on the OntoNotes dataset with GPT-4o, DEER achieved an F1 score of 85.2, significantly higher than the 80.7 achieved by KATE.
  • Parity with Supervised Models: On four out of five datasets, DEER (using GPT-4o) matched or exceeded the performance of specialized, fine-tuned models like BERT-base and UniversalNER-7B.
  • Data Efficiency: The framework proved robust in low-resource settings, maintaining strong performance even with as few as 16 training examples.

Conclusion

DEER demonstrates that the distribution of labels in training data is a powerful, often overlooked signal for in-context learning. By shifting from task-agnostic retrieval to label-guided retrieval and incorporating a targeted reflection step, general-purpose LLMs can achieve state-of-the-art performance on fine-grained tasks like NER. Future work will focus on extending DEER to handle nested and discontinuous entities, as well as automating the detection of error types during the reflection phase.

Summary updated

BibTeX
@misc{bai2025label,
  author = {Fan Bai and Hamid Hassanzadeh and Ardavan Saeedi and Mark Dredze},
  title = {{Label-Guided In-Context Learning for Named Entity Recognition}},
  year = {2025},
  eprint = {2505.23722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.23722},
}
Figure