JRC TMA-CC: Slavic Named Entity Recognition and Linking. Participation in the BSNLP-2019 shared task

Guillaume Jacquet, Jakub Piskorski, Hristo Tanev, Ralf SteinbergerACL 2019

TL;DR

This paper presents a hybrid system for Slavic Named Entity Recognition (NER) and linking, combining rule-based approaches with machine learning. The core innovation lies in a novel method for handling name inflections by learning suffix replacement rules from large text corpora, significantly improving recall while maintaining high precision.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and categorize entities like persons, organizations, and locations in text. However, Slavic languages, with their rich morphology, present a significant challenge due to extensive name inflections. Standard NER systems often struggle to recognize these variants, leading to low recall. This paper tackles the problem of improving NER performance in Slavic languages by effectively handling these morphological variations, specifically focusing on increasing recall without sacrificing the high precision that is crucial for downstream applications.

Methods

The authors developed four NER systems, all built upon their in-house NERONE system, which is known for its high precision but limited recall. The core of their approach involves a multi-stage process:

  1. Initial Recognition: They leverage multilingual lexical resources (JRC-NAMES, BABELNET) and a named entity guesser to identify known names.
  2. Recall Enhancement with Wildcards: To capture inflectional variants, they introduce a wildcard mechanism. This involves removing the last few letters of name parts and replacing them with a wildcard character (e.g., "Muguruzaová" becomes "Muguruzao%"). This significantly boosts recall by matching variations.
  3. Precision Improvement via Suffix Filtering: To mitigate the over-generation caused by wildcards, they implement a sophisticated suffix filtering technique. This involves:
  4. Analyzing large news article collections to identify common suffix replacement patterns for known names.
  5. Learning probabilistic rules for these suffix replacements.
  6. Filtering out potential name matches that do not adhere to these learned, high-probability rules.

They explored different thresholds for this suffix filtering, leading to four distinct system variants (JRC-TMA-CC-1 to JRC-TMA-CC-4).

Datasets

The primary datasets used and generated in this work are:

  • JRC-NAMES: A publicly distributed resource containing multilingual named entity variants and titles.
  • BABELNET: A wide-coverage multilingual semantic network used for disambiguation.
  • GeoNames: A resource used to derive location names.
  • News Article Collections: Large corpora of news articles (approximately 2 million per language) were used to learn suffix replacement rules. The paper mentions collecting data for Czech, Russian, Bulgarian, and Polish.
  • BSNLP-2019 Shared Task Data: The official training and test datasets for the BSNLP-2019 Shared Task were used for evaluation.

Key Insights and Ideas

  • The Power of Learned Suffix Rules: The most significant insight is that learning probabilistic suffix replacement rules from large text corpora is a highly effective way to filter out false positives generated by wildcard-based recall enhancement. This allows for a better precision-recall trade-off.
  • Wildcards as a Recall Booster: The simple yet effective use of wildcards to capture morphological variants is a key enabler for increasing recall in highly inflected languages.
  • Hybrid Approach Pays Off: Combining a robust, high-precision base system (NERONE) with rule-based and statistical methods for handling inflections demonstrates the strength of hybrid approaches in tackling complex NLP challenges.
  • High Precision is Achievable: Despite the focus on increasing recall, the authors consistently achieved very high precision scores across all languages, often ranking first among participants.

Key Results

The paper reports strong performance on the BSNLP-2019 Shared Task, particularly in terms of precision.

  • Precision: The JRC-TMA-CC-4 system achieved an average precision of 94.4% for Czech, 90.2% for Russian, 95.4% for Bulgarian, and 93.7% for Polish. They ranked first among all participants for precision across all four languages.
  • F-measure: While recall was lower than desired for some systems, the F-measure scores were competitive. For example, JRC-TMA-CC-4 achieved F-measures of 58.0% for Czech, 73.5% for Russian, 74.2% for Bulgarian, and 57.4% for Polish on the test data.
  • Entity Linking: For entity linking, their F-measure scores were ranked 2nd for both single-language and cross-lingual linking, indicating good performance despite the challenges.
  • System Performance: The JRC-TMA-CC-2 system (NERONE + wildcards) generally achieved the highest recall, while JRC-TMA-CC-3 and JRC-TMA-CC-4 showed a better balance between precision and recall due to suffix filtering.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Recall Improvement: While recall was improved, it could still be further enhanced.
  • Wildcard Over-generation: The wildcard mechanism, even with filtering, can still lead to over-generation and errors.
  • Coverage of Geographical Names: The coverage of geographical names is inconsistent across languages, ranging from over 100,000 to below 3,000.
  • Future Work Suggestions:
    • Expanding the set of learned inflection patterns.
  • Integrating a classifier to distinguish between entities that can have multiple interpretations (e.g., "BBC" as an organization or product).
  • Improving the lookup of geographical names.
  • Developing mechanisms to better distinguish between grammatical case markers and gender markers in languages like Czech.
  • Introducing additional heuristics to refine name mention matches.
  • Updating and completing the list of geographical names.

Summary updated

BibTeX
@inproceedings{jacquet2019jrc,
  author = {Guillaume Jacquet and Jakub Piskorski and Hristo Tanev and Ralf Steinberger},
  title = {{JRC TMA-CC: Slavic Named Entity Recognition and Linking. Participation in the BSNLP-2019 shared task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure