GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition

Shizhou Huang, Bo Xu, Yang Yu, Changqun Li, Xin Alex LinAAAI 2025 · OpenReview

TL;DR

This paper introduces GuideNER, a novel approach to Named Entity Recognition (NER) using Large Language Models (LLMs) with in-context learning (ICL). Instead of relying on input-output examples, GuideNER leverages LLMs to summarize concise annotation guidelines, which are then used as contextual information to improve NER performance. This method significantly outperforms existing ICL techniques, especially on challenging datasets like GENIA.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown remarkable capabilities in various tasks through in-context learning (ICL), where they learn from a few provided examples. However, for Named Entity Recognition (NER), there's a noticeable performance gap compared to traditional fine-tuning methods. The authors hypothesize this stems from inconsistencies in how entities are defined across different NER datasets (e.g., whether pronouns are labeled as "Person"). Crucially, recent research suggests LLMs don't deeply learn the specific input-label mappings from examples; they grasp the general task. This makes example-based ICL less effective for NER, as it fails to explicitly convey the precise scope of each entity type. Existing annotation guidelines can be excessively long, making them impractical for direct use in prompts.

Methods

GuideNER proposes a three-step process to generate and utilize annotation guidelines for ICL-based NER:

  1. Summarize Patterns: An LLM is prompted with a task description, examples, and input-label pairs from the training set. Its goal is to infer and summarize generic rules or patterns for each named entity category in a structured format (e.g., JSON).
  2. Self-Verify: The summarized patterns are then fed back into an LLM along with input text. The LLM predicts labels based on these patterns. These predictions are compared against the ground truth labels. Only patterns that accurately predict entities are retained, filtering out incorrect or ambiguous ones.
  3. Aggregate Rules: The validated rules are aggregated, their frequencies counted, and then filtered to retain the most general and frequent ones. These are condensed into concise annotation guidelines. Finally, these guidelines, along with a task description and examples of their usage, form a "Predict Prompt" that is fed to an LLM for inferring entities on new, unseen text.

This approach contrasts with traditional ICL by using static, condensed guidelines rather than dynamically retrieving examples.

Datasets

The paper evaluates GuideNER on four widely used NER datasets:

  • CONLL03: A news domain dataset with four entity types: Person, Organization, Location, and Miscellaneous.
  • ACE04 & ACE05: News domain datasets with seven entity types, including Person, Organization, Geopolitical entity, and Location.
  • GENIA: A nested NER dataset from the molecular biology domain, featuring five entity types: DNA, RNA, Protein, Cell_type, and Cell_line.

Key Insights and Ideas

  • Guidelines over Examples for NER: The core insight is that for NER, explicitly defining entity categories through annotation guidelines is more effective than relying on input-output examples for in-context learning. This is because guidelines directly address the ambiguity and inconsistency in entity definitions that plague NER.
  • LLMs as Guideline Summarizers: The paper demonstrates that LLMs can effectively summarize complex annotation guidelines into concise, usable formats, making them practical for ICL.
  • Self-Verification is Crucial: The self-verification step is vital for filtering out erroneous or ambiguous patterns generated by the LLM, preventing them from contaminating the final annotation guidelines and degrading performance.
  • Efficiency Gains: GuideNER achieves competitive or superior performance with significantly shorter prompts compared to methods that rely on numerous examples, leading to potential cost and speed advantages.

Key Results

  • GuideNER consistently and significantly outperforms state-of-the-art ICL methods across all four tested datasets.
  • On the GENIA dataset, GuideNER achieved a remarkable 12.63 F1 score improvement over the previous state-of-the-art model.
  • When using text-davinci-003 as the LLM, GuideNER achieved an F1 score of 90.27 on CONLL03, outperforming other methods.
  • The self-verification step was shown to be critical, with performance dropping significantly when it was removed (e.g., a drop from 66.71 to 60.12 on CONLL03 for llama3-8B-Instruct).
  • The method shows robustness to the amount of training data used, with performance stabilizing when using as little as 50-70% of the training set.

Limitations and Future Work

  • Dynamic Rule Retrieval Challenges: While the paper explores using fewer rules (top-k), it notes that current semantic-based retrieval methods struggle to effectively match rules to input text due to semantic gaps, leading to poor performance in retrieval-based approaches.
  • Model Dependency: The performance of GuideNER is influenced by the LLM used for summarization and inference. More advanced LLMs generally yield better results.
  • Future Work: The authors suggest that future research could focus on improving dynamic rule retrieval to better leverage relevant rules for prediction and reduce prompt length further. They also highlight the potential for using fewer rules to achieve comparable performance, which could be beneficial in resource-constrained environments.

Summary updated

Abstract

Large language models (LLMs) demonstrate impressive performance on downstream tasks through in-context learning(ICL). However, there is a significant gap between their performance in Named Entity Recognition (NER) and in fine-tuning methods. We believe this discrepancy is due to inconsistencies in labeling definitions in NER. In addition, recent research indicates that LLMs do not learn the specific input-label mappings from the demonstrations. Therefore, we argue that using examples to implicitly capture the mapping between inputs and labels in in-context learning is not suitable for NER. Instead, it requires explicitly informing the model of the range of entities contained in the labels, such as annotation guidelines. In this paper, we propose GuideNER, which uses LLMs to summarize concise annotation guidelines as contextual information in ICL. We have conducted experiments on widely used NER datasets, and the experimental results indicate that our method can consistently and significantly outperform state-of-the-art methods, while using shorter prompts. Especially on the GENIA dataset, our model outperforms the previous state-of-the-art model by 12.63 F1 scores.
BibTeX
@inproceedings{huang2025guidener,
  author = {Shizhou Huang and Bo Xu and Yang Yu and Changqun Li and Xin Alex Lin},
  title = {{GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=RdXntUjrWs},
}
Figure