Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French

Alice Millour, Yoann Dupont, Karen Fort, Liam DuignanCOLING 2024

TL;DR

This paper tackles the challenge of fairly evaluating Named Entity Recognition (NER) systems in French by introducing a comprehensive, multi-genre gold corpus and a flexible evaluation framework. By accounting for variations in annotation schemes and textual genres, their approach enables a more robust comparison of NER system performance, revealing strengths and weaknesses that simpler evaluations might miss.

What Problem Does This Paper Address?

Named Entity Recognition (NER) systems are crucial for extracting structured information from text, but their performance is notoriously difficult to compare. This difficulty stems from two main issues:

  1. Varying Annotation Schemes: Different NER systems use their own unique "tagsets" (the categories of entities they identify, like Person, Organization, Location). These tagsets can differ significantly in their precision (how detailed they are) and coverage (how many types of entities they can identify). This makes it hard to directly compare how well two systems perform if they're looking for slightly different things.
  2. Corpus Homogeneity and Robustness: Standard evaluation often relies on homogeneous corpora (texts from similar sources or genres). However, NER systems can be brittle and perform poorly on text that differs from their training data. An evaluation on a narrow corpus might not accurately reflect a system's real-world performance across diverse text types.

The authors aim to overcome these limitations by creating a richer evaluation framework that considers both the comparability of annotation schemes and the adequacy of systems across different textual genres.

Methods

The core of the paper's contribution lies in its evaluation methodology and the creation of a new resource:

  • Rich Evaluation Framework: The authors propose a framework with three dimensions: the precision of the tagset, the textual genres of the evaluation corpora, and the evaluation metrics. This allows for a more nuanced understanding of system performance.
  • Gold Corpus Creation: They developed a new, freely available gold corpus for French. This corpus is designed to be balanced across six textual genres (prose, poetry, speech, encyclopedic, multisource, and informative) and is annotated with a rich tagset (8 types and 23 subtypes) derived from the Quæro tagset. This richness allows it to serve as a reference for multiple existing annotation schemes.
  • Annotation Scheme Mapping: To enable comparison between systems with different tagsets, the authors created intermediate "meta-tagsets." These mappings allow for evaluating systems on their native tagsets, on a common minimal tagset (like the one used in the CoNLL-2003 shared task), and on a meta-tagset that bridges the gap between their rich gold standard and the systems' tagsets.
  • System Evaluation: Four freely available French NER systems were evaluated: one rule-based (CasEN) and three neural network-based (SpaCy, Flair, and CamemBERT-ner). They were evaluated both individually on their target tagsets and comparatively on common meta-tagsets.

Datasets

  • Newly Introduced Benchmark: The paper introduces a new gold corpus for French NER.
  • Size: 15 samples, each approximately 1,000 tokens, totaling 15,200 tokens.
  • Genres: 6 textual genres (prose, poetry, speech, encyclopedic, multisource, informative).
  • Annotation: Annotated with a rich tagset (8 types, 23 subtypes) based on the Quæro tagset.
  • Availability: Freely available.
  • Existing Datasets (for system training/prior evaluation):
    • ESLO 1 corpus (used for CasEN evaluation).
  • ESTER 2 evaluation campaign (used for CasEN evaluation).
  • WikiNER (used for training/fine-tuning SpaCy, Flair, and CamemBERT-ner).

Key Insights and Ideas

  • Annotation Scheme is a Dimension: The authors emphasize that the annotation scheme itself is a critical dimension of evaluation, not just a fixed target. Understanding how systems map to different schemes is crucial for fair comparison.
  • Genre Matters for Robustness: Evaluating systems across diverse genres reveals their robustness. Some systems perform consistently well across genres, while others show significant fluctuations, indicating potential overfitting to specific text types.
  • Richness vs. Simplicity: A richer gold standard (like their Quæro-based one) allows for more detailed analysis and comparison. However, mapping this richness to simpler, common tagsets (like the minimal CoNLL-2003 tagset) leads to a loss of information and a reduction in the number of identifiable entities.
  • Error Profiles Can Be Similar: Despite differences in performance, the types of errors made by different systems can be surprisingly similar, suggesting common challenges in NER.
  • CasEN's Stability: The rule-based CasEN system showed stable performance across literary genres, contrasting with the more variable performance of neural systems on these genres.

Key Results

  • CamemBERT-ner Outperforms Others: Globally, CamemBERT-ner achieved the highest F-scores among the evaluated systems.
  • Genre-Specific Performance Differences: The paper highlights that CasEN outperformed Flair and SpaCy on literary genres, while these neural systems performed better on "Multisource" and "Encyclopedic" genres.
  • Labeling vs. Segmentation: While CasEN might not have the highest overall F-score, it showed strong performance on the labeling subtask, indicating its strength in classifying entities once segmented.
  • Loss of Entities in Mapped Corpora: When mapping the rich gold corpus to simpler tagsets (like Meta-WikiNER), entities like "amount" and "date" were excluded, and "prod" annotations were merged into "MISC," demonstrating the information loss in coarser schemes.

Limitations and Future Work

  • Corpus Size and Balance: The current gold corpus, while rich, is still relatively small (15,200 tokens) and has under-represented genres like poetry and encyclopedia. The authors plan to extend the corpus and improve its genre balance.
  • Finer-Grained Analysis: The authors suggest future work could involve a finer per-type analysis of errors and the use of more interpretable metrics that consider entity attributes like length, frequency, and ambiguity, as proposed by Fu et al. (2020).
  • Overfitting Tendencies: The study confirms that challenging systems on diverse corpora can reveal over-fitting tendencies, suggesting that future evaluations should prioritize broad textual coverage.

Summary updated

BibTeX
@inproceedings{millour2024unveiling,
  author = {Alice Millour and Yoann Dupont and Karen Fort and Liam Duignan},
  title = {{Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure