Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model

Subhadip Nandi, Neeraj AgrawalEMNLP 2024

TL;DR

This paper introduces IF-WRANER, a novel approach to Few-Shot Cross-Domain Named Entity Recognition (NER) that leverages instruction-tuned open-source Large Language Models (LLMs) with a retrieval-augmented generation framework. By using word-level embeddings for retrieval and applying specific regularization techniques, IF-WRANER achieves state-of-the-art performance while being more cost-effective and adaptable than previous methods.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in information extraction, aiming to identify and categorize entities (like people, organizations, locations) within text. A significant challenge arises when applying NER models to new domains where labeled data is scarce – this is known as Few-Shot Cross-Domain NER. Traditional approaches often require extensive fine-tuning on target domain data, which is computationally expensive and impractical for real-world scenarios. While recent work has explored Large Language Models (LLMs) for this task, many rely on proprietary models like GPT-4, which are costly, or struggle to precisely follow instructions, especially with open-source alternatives. This paper addresses the need for a cost-effective, adaptable, and high-performing solution for Few-Shot Cross-Domain NER using readily available open-source LLMs.

Methods

The core of IF-WRANER lies in its innovative combination of several techniques:

  • Instruction Tuning: Instead of traditional fine-tuning for domain adaptation, the authors instruction-tune an open-source LLM (specifically, a 7B Meta LLM) on a source domain dataset. The goal isn't to learn the source domain's specifics, but rather to teach the LLM how to perform NER and adhere to structured output formats as defined in prompts.
  • Retrieval Augmented Generation (RAG): To provide task-specific context during inference, a RAG framework is employed. This involves storing labeled examples from the target domain in a vector database. When a new query arrives, relevant examples are retrieved based on similarity and prepended to the prompt fed to the LLM.
  • Word-Level Embedding for Retrieval: A key innovation is the use of word-level embeddings for retrieving relevant examples, rather than the more common sentence-level embeddings. The authors argue that for NER, which is inherently a word-level task, word-level similarity provides more pertinent examples, leading to better performance.
  • Regularization Techniques: To combat overfitting during the instruction tuning phase and ensure the LLM adheres to prompt instructions, several regularization techniques are applied. These include duplicating training examples with randomly removed entity types and shuffling the order of entity types in prompts.

Datasets

The paper uses the CrossNER dataset for evaluating target domains. This dataset comprises five diverse domains: politics, natural science, music, AI, and literature. For the source domain, the authors utilize a subset of the CoNLL 2003 dataset, a common practice in Cross-Domain NER research. The authors also mention developing their model for proprietary customer care domains of an enterprise, but these datasets are not shared due to their proprietary nature.

Key Insights and Ideas

  • Word-Level Retrieval is Crucial for NER: The finding that word-level embeddings significantly outperform sentence-level embeddings for retrieval in NER tasks is a major insight. This highlights the importance of matching granular semantic units for tasks that focus on local context.
  • Instruction Tuning for Adaptability: Shifting the focus from domain-specific fine-tuning to instruction tuning for prompt adherence makes the model highly adaptable. This means the same base model can be used across many domains with minimal effort, simply by providing new entity definitions and a few examples.
  • Open-Source LLMs Can Compete: The paper demonstrates that well-tuned open-source LLMs, when combined with effective RAG and instruction following, can achieve performance comparable to or even exceeding proprietary models like GPT-4, making advanced NER solutions more accessible.
  • Regularization is Key for Robustness: The ablation study clearly shows that regularization techniques are vital for preventing the LLM from memorizing specific entity types and ensuring it consistently follows the provided instructions, especially in the presence of ambiguous entity definitions.

Key Results

IF-WRANER achieves state-of-the-art performance on the CrossNER benchmark, showing a more than 2% F1 score improvement over previous SOTA models. Specifically, it outperforms other non-LLM approaches and is competitive with PromptNER using GPT-4. The model also demonstrates significant cost savings and reduced latency compared to GPT-4 based solutions, especially when using a smaller LLM (Tinyllama) for low-latency requirements. In a real-world enterprise deployment for customer care domains, IF-WRANER helped reduce escalations to human agents by almost 15%, leading to substantial yearly savings.

Limitations and Future Work

The authors acknowledge that while their model achieves strong performance, there can be issues with overfitting, which they address with regularization. They also note that for extremely low-latency requirements, a smaller LLM (Tinyllama) is used, which results in a slight drop in F1-score. Future work could explore more advanced regularization techniques or further optimize the RAG component for even greater efficiency and accuracy. The paper also suggests that the model's behavior can be influenced by the order of entity type definitions in prompts, indicating potential avenues for further prompt engineering research.

Summary updated

BibTeX
@inproceedings{nandi2024improving,
  author = {Subhadip Nandi and Neeraj Agrawal},
  title = {{Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure