Abstract
Named Entity Recognition (NER) is a critical task for identifying specific tokens within text, yet Large Language Models (LLMs) often struggle to match the performance of specialized, fine-tuned models when using standard In-Context Learning (ICL). This paper introduces DEER (Data stat.-grounded namE Entity Recognition), a training-free framework that leverages label-grounded statistics to guide both demonstration retrieval and a targeted error-reflection process. By prioritizing relevant entity and context patterns, DEER significantly boosts NER performance across diverse benchmarks, achieving results comparable to supervised fine-tuning without requiring model updates.
Introduction
Named Entity Recognition (NER) requires precise, token-level identification of entities such as names, locations, or medical terms. While LLMs excel at general language tasks, their performance on NER via ICL is often limited by suboptimal demonstration retrieval. Most existing methods rely on task-agnostic semantic similarity, selecting examples that sound like the test query but may contain entirely different entity patterns. Furthermore, standard ICL typically follows a single-pass inference process, leaving no room for the model to correct errors on ambiguous or rare tokens.
To address these bottlenecks, we propose DEER, which utilizes training set labels as a signal rather than just a target. By analyzing the distribution of labels across tokens, DEER identifies the most relevant demonstrations and implements a reflection step to refine initial predictions.
Methodology
DEER is a three-stage, training-free framework designed to maximize the utility of available training data during inference.
1. Data Statistics Preparation
Before inference, the framework analyzes the training set to compute frequencies and probabilities for three token categories: entity tokens (part of a label), context tokens (words immediately surrounding an entity), and other tokens. This statistical profile serves as the foundation for the subsequent retrieval and reflection steps.
2. Label-Guided Demonstration Retrieval
Unlike traditional methods that use sentence-level embeddings, DEER uses pre-computed statistics to prioritize training examples that share relevant entity or context patterns with the test query. This ensures the LLM is provided with demonstrations that are contextually and structurally relevant to the specific entities it must identify.
3. Error Reflection
Following an initial prediction, DEER identifies "error-prone" tokens—such as unseen words in entity-rich contexts or tokens frequently labeled as entities in training but missed in the current pass. The model is then prompted to reflect on these specific tokens using targeted, span-level demonstrations to produce an updated, more accurate prediction.
Experimental Results
DEER was evaluated across five diverse datasets (CoNLL03, OntoNotes, NCBI-disease, BC2GM, and TweetNER7) using state-of-the-art LLMs including Qwen2.5, Llama3.3, and GPT-4o.
- Performance Gains: DEER consistently outperformed standard ICL baselines. For example, on the OntoNotes dataset with GPT-4o, DEER achieved an F1 score of 85.2, significantly higher than the 80.7 achieved by KATE.
- Parity with Supervised Models: On four out of five datasets, DEER (using GPT-4o) matched or exceeded the performance of specialized, fine-tuned models like BERT-base and UniversalNER-7B.
- Data Efficiency: The framework proved robust in low-resource settings, maintaining strong performance even with as few as 16 training examples.
Conclusion
DEER demonstrates that the distribution of labels in training data is a powerful, often overlooked signal for in-context learning. By shifting from task-agnostic retrieval to label-guided retrieval and incorporating a targeted reflection step, general-purpose LLMs can achieve state-of-the-art performance on fine-grained tasks like NER. Future work will focus on extending DEER to handle nested and discontinuous entities, as well as automating the detection of error types during the reflection phase.
Summary updated