CALM: Context Augmentation with Large Language Model for Named Entity Recognition

Tristan Luiggi, Tanguy Herserant, Thong Tran, Laure Soulier, Vincent Guigue

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, involving the identification and categorization of entities such as organizations, persons, and locations. However, NER models frequently encounter challenges related to data scarcity and semantic ambiguity. For instance, the word "Apple" in a brief sentence could refer to either a fruit or a technology corporation. Traditional approaches have attempted to resolve this by retrieving external context from search engines, but these methods are often noisy, costly, and difficult to control.

To address these limitations, the CALM (Context Augmentation with Large Language Model) framework leverages the generative capabilities of Large Language Models (LLMs) to provide rich, task-specific context for ambiguous sentences, creating a more self-contained and effective NER system.

Methodology

The CALM framework splits the NER process into two primary modules: the Augmentor and the Tagger.

The CALM architecture featuring the Augmentor and Tagger modules.

The Augmentor (LLM)

Instead of relying on external search APIs, CALM uses Llama 2-7B to generate context. The effectiveness of the LLM is maximized through a systematic prompting framework that defines both the task and the variation of the prompt.

Components of the systematic prompting framework used for context generation.

The prompting strategy is divided into two main components:

  • Task Selection: Strategies include Contextualization (asking for facts about entities), Reformulation (rephrasing the input), and Contextual Variability (placing entities in different scenarios).
  • Variation "Flavors": Prompts are enhanced with Personas (e.g., "act as a linguist"), Reflection Patterns (asking the LLM to address its own potential ambiguities), and specific Answer Formats.

The Tagger and Cooperative Learning

The Tagger is a transformer-based model (such as RoBERTa) that processes the original sentence alongside the LLM-generated context to predict entity tags. To ensure practical deployment, the authors employ Cooperative Learning. This technique trains the Tagger to mimic the performance of the augmented model even when the LLM context is unavailable during inference, effectively distilling the LLM's knowledge into the smaller tagger and reducing computational overhead in production environments.

Evaluation and Key Insights

CALM was evaluated across three diverse benchmarks: WNUT17 (noisy social media text), BC5CDR (biomedical data), and CoNLL++ (news articles).

Performance and State-of-the-Art Results

CALM achieved new state-of-the-art (SOTA) F1 scores on WNUT17 (61.54) and CoNLL++ (96.00), outperforming previous search-engine-based baselines. The LLM-generated contexts proved more informative than search results; for example, 60% of input entities were present in the LLM context, compared to only 44% in Google Search results.

Prompting as a Quality Filter

Qualitative analysis revealed that "steering" the LLM is critical. Combining all prompt variations (Persona + Reflection + Format) significantly increased the rate of "Correct" context generations. Furthermore, the study found that complementarity is more important than similarity. Prompts that provided additional, unique information outperformed those that merely paraphrased the input.

Specialized Domains

While CALM excelled in general domains, it faced hurdles in the biomedical sector (BC5CDR). General-purpose LLMs like Llama 2 struggle with highly specialized medical jargon without domain-specific fine-tuning, suggesting a need for specialized augmentors in such fields.

Conclusion

CALM demonstrates that LLMs can serve as effective, self-contained knowledge bases for context augmentation in NER. By replacing noisy search-engine retrieval with structured prompting, the framework achieves superior performance on major benchmarks. Future work will explore "soft prompting" to automatically optimize LLM queries and investigate the use of smaller, more efficient models to further reduce computational costs.

Summary updated

Abstract

In prior research on Named Entity Recognition (NER), the focus has been on addressing challenges arising from data scarcity and overfitting, particularly in the context of increasingly complex transformer-based architectures. A framework based on information retrieval (IR), using a search engine to increase input samples and mitigate overfitting tendencies, has been proposed. However, the effectiveness of such system is limited, as they were not designed for this specific application. While this approach serves as a solid foundation, we maintain that LLMs offer capabilities surpassing those of search engines, with greater flexibility in terms of semantic analysis and generation. To overcome these challenges, we propose CALM an innovative context augmentation method, designed for adaptability through prompting. In our study, prompts are meticulously defined as pairs comprising specific tasks and their corresponding response strategies. This careful definition of prompts is pivotal in realizing optimal performance. Our findings illustrate that the resultant context enhances the robustness and performances on NER datasets. We achieve state-of-the-art F1 scores on WNUT17 and CoNLL++. We also delve into the qualitative impact of prompting.
BibTeX
@inproceedings{luiggi2024calm,
  author = {Tristan Luiggi and Tanguy Herserant and Thong Tran and Laure Soulier and Vincent Guigue},
  title = {{CALM: Context Augmentation with Large Language Model for Named Entity Recognition}},
  booktitle = {TPDL},
  year = {2024},
  doi = {10.1007/978-3-031-72437-4\_16},
}
Figure