UkraiNER: A New Corpus and Annotation Scheme towards Comprehensive Entity Recognition

Lauriane Aufrant, Lucie ChasseurCOLING 2024

TL;DR

This paper introduces UkraiNER, a novel annotation scheme and corpus for French Named Entity Recognition (NER) that goes beyond traditional "flat" NER to capture nested, discontinuous, and non-named entities. This comprehensive approach aims to unlock richer information extraction for real-world applications, and the authors release both the annotation tool and the UkraiNER corpus to facilitate future research.

What Problem Does This Paper Address?

Traditional Named Entity Recognition (NER) systems have historically focused on identifying simple, non-overlapping mentions of proper names (e.g., "Paris," "John Doe"). However, this approach misses a significant amount of valuable information in real-world text. For instance, it fails to capture entities like "students" or "NATO representative" in the example sentence, or nested entities like "the French President Jacques Chirac" (which contains "the French President" and "Jacques Chirac"). While some previous efforts have attempted to broaden NER's scope to include nested or non-named entities, these annotation schemes have often been complex and have seen limited adoption. Furthermore, there's a notable scarcity of comprehensive, manually annotated data for French compared to English. UkraiNER aims to bridge this gap by providing a more comprehensive annotation scheme and a corresponding corpus that is both rich in information and relatively simple to use.

Methods

The core contribution of this paper is a new annotation scheme designed for "Comprehensive Entity Recognition" (CER). This scheme is built on three main principles:

  1. Broad Entity Coverage: It encompasses a wide range of entity types, including Persons, Locations, Dates, Events, Organizations, Groups, Products, and Others. Crucially, it uses coarse types, avoiding fine-grained distinctions (e.g., treating all organizations similarly).
  2. Full Mention Annotation: All mentions of entities within these classes are annotated, including nested, non-named, and discontinuous entities. This means if "students" is an entity, every instance of "students" referring to that entity is marked.
  3. Rich Information for Inclusions: Entities that might be excluded by simpler schemes are annotated with rich metadata. For example, each entity is labeled as "named" or "non-named," providing flexibility for downstream tasks.

The authors also developed a lightweight, portable annotation tool to implement these guidelines and visualize annotated corpora. The annotation format extends the CoNLL-U format, incorporating entity information as inline comments above tokens, creating a "CoNLL-UI" format.

Datasets

The paper introduces the UkraiNER corpus, a new benchmark dataset consisting of 10,000 French sentences extracted from news briefs from the newspaper Le Monde, focusing on the Ukrainian situation in February-March 2022. This corpus is manually annotated using the proposed comprehensive entity recognition guidelines. The data was collected from online live feeds and pre-processed using UDPipe. For evaluation, the UkraiNER corpus is split into training, development, and test sets.

Key Insights and Ideas

  • Beyond Proper Nouns: The paper strongly advocates for moving beyond traditional NER's focus on proper nouns. Annotating non-named entities (like "students" or temporal expressions like "earlier") is crucial for capturing a more complete understanding of text.
  • Nested Entities are Key: The authors demonstrate how nested entities provide essential context. For example, recognizing "the French President" as a nested entity within "the French President Jacques Chirac" is vital for understanding the specific role and individual.
  • Flexibility through Metadata: Labeling entities as "named" or "non-named" offers a practical way to balance comprehensiveness with the ability to filter for traditional NER tasks if needed, making the scheme adaptable.
  • Challenges of Comprehensiveness: The baseline experiments reveal that comprehensive entity recognition is significantly more challenging than traditional nested NER, particularly for non-named entities. This highlights the need for new models and approaches.

Key Results

The UkraiNER corpus contains 10,604 sentences with 43,623 annotated entities. Baseline experiments using a state-of-the-art nested NER model (CNN-NER) show that:

  • Comprehensive NER is Harder: Training CNN-NER on the full UkraiNER annotations (CER) resulted in a lower F1 score (82.3) compared to training on a reduced set that mimics traditional nested NER (NNER, F1 of 87.4). This indicates the increased difficulty of the comprehensive task.
  • Traditional Models Struggle: When trained on the comprehensive annotations but evaluated on a filtered set mimicking traditional nested NER (NNER→CER), the model had low recall, failing to identify over half of the entities.
  • Non-Named Entities are a Bottleneck: The recall for non-named entities was significantly lower (68.6) than for named entities (96.5) when using the comprehensive annotation scheme, underscoring their difficulty.
  • Comparison to Existing Work: The CER→CER F1 score (82.3) is comparable to existing work on the Genia corpus, while the NNER→NNER F1 (87.4) aligns with results on ACE datasets, suggesting the baseline models are performing reasonably well within their respective paradigms.

Limitations and Future Work

The authors acknowledge several limitations:

  • Discontinuous Entities: The current scheme identifies discontinuous entities but does not fully annotate all their constituent tokens, marking them as "incomplete." This is a known area of active research.
  • Single Annotator Bias: The corpus was primarily annotated by a single individual, which could introduce biases. This was partially mitigated by quality checks and counter-annotations.
  • Future Work: The authors suggest adapting the guidelines for user-generated content (like tweets) and complementing the entity annotations with other information extraction layers, such as coreference resolution, entity linking, and relation extraction, to build more robust knowledge bases. They also propose making the guidelines interoperable with existing entity linking approaches.

Summary updated

BibTeX
@inproceedings{aufrant2024ukrainer,
  author = {Lauriane Aufrant and Lucie Chasseur},
  title = {{UkraiNER: A New Corpus and Annotation Scheme towards Comprehensive Entity Recognition}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure