Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset

Arthur Amalvy, Vincent Labatut, Richard DufourEMNLP 2023 · DOI

Abstract

Standard Transformer-based Named Entity Recognition (NER) models often struggle with long-form documents due to limited context windows. This paper presents a method to train a neural re-ranker using synthetic data generated by an LLM (Alpaca). This re-ranker identifies helpful context from across an entire document, which is then used to augment the input sentence. Experiments on literary datasets demonstrate that this approach significantly outperforms unsupervised retrieval methods, providing a substantial boost in NER performance by enabling long-range disambiguation.

Introduction

Named Entity Recognition (NER) is a foundational task in NLP, but its performance often degrades in long-form literature. Standard models like BERT have a limited context window, making it difficult to disambiguate entities that require information from hundreds of pages away. For instance, a character introduced in Chapter 1 might only be identifiable as a specific person through a description provided in Chapter 10. While retrieval-augmented generation (RAG) techniques can help, there is a lack of supervised datasets that specify which sentences are truly helpful for NER disambiguation.

Methodology

To address the lack of supervised data, the authors propose a Synthetic Data Generation framework. Instead of manual annotation, they utilized Alpaca (an instruction-tuned Llama model) to generate helpful context sentences for specific entities.

The Pipeline

As illustrated in the architecture below, the system operates in several stages:

  1. Retrieval Heuristic: A broad pool of candidate sentences is gathered from the entire document using simple methods like BM25 or noun matching.
  2. Neural Re-ranking: A BERT-based regression model, trained on the synthetic dataset, scores these candidates for relevance.
  3. Context Augmentation: The top $k$ sentences are concatenated to the original input sentence.
  4. NER Inference: The final augmented string is passed to a standard NER model for classification.
The proposed pipeline for context-augmented NER.

Datasets

The study utilized three primary data sources:

  • English Literary Dataset: Comprising the first chapters of 40 novels, used for primary evaluation.
  • Synthetic Context Retrieval Dataset: A new dataset containing ~2,700 examples generated by Alpaca-7b and Alpaca-13b to provide supervision for the re-ranker.
  • MSMarco: Used as a baseline to compare the synthetic-trained model against general-purpose human-annotated retrieval data.

Key Results and Discussion

The proposed neural re-ranker consistently outperformed unsupervised baselines and heuristics. Key findings include:

  • Performance Gain: The system achieved an F1-score improvement of approximately 1 point over the baseline without retrieval.
  • Optimal Context ($k$): The best results were typically observed when retrieving exactly 3 context sentences ($k=3$). As shown in the performance plots, adding too much context (increasing $k$ beyond 4 or 5) can introduce noise that degrades model performance.
  • Synthetic vs. Human Data: The re-ranker trained on LLM-generated synthetic data performed on par with models trained on massive human-annotated datasets like MSMarco, suggesting that task-specific synthetic data is highly effective.
NER performance (F1-score) across different values of k and candidate pool sizes.

Conclusion

This research demonstrates that LLMs can be effectively used as "knowledge proxies" to generate synthetic supervision for specialized NLP tasks. By training a neural re-ranker on this data, NER models can effectively "look back" through long documents, overcoming the inherent context limitations of the Transformer architecture. Future work will focus on reducing computational overhead and ensuring diversity in the retrieved context sentences.

Summary updated

BibTeX
@inproceedings{amalvy2023learning,
  author = {Arthur Amalvy and Vincent Labatut and Richard Dufour},
  title = {{Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.642},
}
Figure