Creating Ontology-annotated Corpora from Wikipedia for Medical Named-entity Recognition

Johann Frei, Frank KramerACL 2024

TL;DR

This paper presents a novel, unsupervised method for creating medical named-entity recognition (NER) corpora by leveraging Wikipedia and Wikidata. By using Wikidata's ontology to define annotation rules, they generate weakly annotated data that is then refined into a "silver standard" corpus, demonstrating competitive results on German medical NER tasks. This approach significantly lowers the barrier to creating specialized NLP resources, especially for low-resource languages.

What Problem Does This Paper Address?

Acquiring high-quality annotated corpora for medical Natural Language Processing (NLP) is a significant challenge. This is due to strict privacy regulations, the high cost and effort of manual annotation, and the fact that existing public datasets may not align with the specific needs of a target application in terms of language, annotation style, or domain. For instance, a dataset annotated for general English might not be suitable for German clinical text. Furthermore, even when label classes appear similar across datasets, the underlying annotation guidelines can differ, making direct comparison and transfer difficult. Manually creating custom annotation layers for specific needs is often infeasible due to its cost and potential for inconsistency.

Methods

The core of this work is an unsupervised approach to generate annotated corpora. The authors combine Wikipedia and Wikidata, treating Wikipedia as a source of text and Wikidata as a structured knowledge base.

  1. Mapping Wikipedia and Wikidata: They establish a mapping between Wikidata entities (language-independent concepts with unique QID numbers) and their corresponding language-specific Wikipedia pages. This is crucial for linking structured knowledge to unstructured text.
  2. Extracting Annotations from Wikipedia: They parse Wikipedia's WikiText markup to identify references (hyperlinks) within sentences. Sentences containing references to concepts of interest (defined by a Wikidata query) are extracted. The text spans of these links and their target pages are recorded. References to concepts not in the desired set are treated as "negative mentions." This process yields a weakly annotated corpus, meaning not every relevant entity is annotated.
  3. Graph-defined Entity Selection using SPARQL: To define the set of concepts for annotation, they leverage SPARQL, a query language for RDF data, to query Wikidata. This allows for complex, ontology-driven selection of entities. For example, they filter for Wikidata entities with assigned ATC (Anatomical Therapeutic Chemical) codes to target medication-related entities.
  4. Dataset Imputation of Weak Annotations: To overcome the limitations of weak annotations (low recall), they employ a dynamic token loss scaling technique during the training of a first-stage NER model. This method assigns different weights to positive, negative, and unknown tokens, allowing the model to learn from the noisy data.
  5. Second-stage NER on Imputed Dataset: The fine-tuned first-stage NER model is then used in inference mode on the weakly annotated corpus to generate "silver standard" annotations. This fully annotated corpus is then used to train a final NER model, often using standard tools like SpaCy.

Datasets

  • Training/Evaluation Data: The primary dataset generated by the authors is a weakly annotated corpus derived from German Wikipedia and Wikidata, filtered using SPARQL queries (specifically targeting entities with ATC codes for medication detection).
  • External Evaluation Datasets: The performance of their trained NER models was evaluated on several existing German medical NER datasets:
  • BRONCO150
    • CARDIO:DE
    • GPTNERMED
    • GERNERMED++
    • GGPOnc2

Key Insights and Ideas

  • Leveraging Linked Data for NLP: The paper demonstrates a powerful synergy between unstructured text (Wikipedia) and structured knowledge graphs (Wikidata) for creating specialized NLP datasets. This approach moves beyond simple text mining to incorporate semantic relationships.
  • Ontology-driven Annotation Rules: Using Wikidata's ontology to define annotation rules via SPARQL queries offers a flexible and explainable way to specify what constitutes a "named entity" for a given task, allowing for custom annotation guidelines without manual effort.
  • Dynamic Loss Scaling for Weak Supervision: The proposed dynamic loss scaling mechanism is a clever way to mitigate the impact of noisy or incomplete labels in weakly supervised settings, effectively balancing precision and recall.
  • Unsupervised Creation of "Silver Standard" Data: The two-stage imputation process, where a model trained on weakly labeled data is used to create a more complete, albeit "silver," dataset for final training, is a significant contribution to reducing annotation bottlenecks.
  • Openness and Reproducibility: The entire pipeline relies on open models and data resources, making the approach reproducible and promoting the sharing of models and corpora.

Key Results

The authors report competitive results on several German medical NER datasets. For instance, their approach achieved F1 scores comparable to or exceeding baseline models on datasets like BRONCO150 and GERNERMED++. They observed that the dynamic loss scaling significantly improves recall at the expense of minor precision losses, with optimal performance often seen at lower Wunk values (indicating better handling of unknown tokens). While performance varied across datasets, particularly lower on GGPOnc2, the overall success on external benchmarks validates the effectiveness of their unsupervised, ontology-driven approach.

Limitations and Future Work

  • Conceptual Disagreements: The authors acknowledge that even with their method, conceptual disagreements can arise. For example, a term might be linked to an ATC code in Wikidata but not annotated in a gold standard dataset, or vice-versa.
  • Annotation Granularity: The method's ability to capture certain types of entities (e.g., strength or frequency of medication) might be limited by the coverage of Wikipedia and Wikidata.
  • Pre-trained Language Models: The quality of the underlying pre-trained language models (like GottBERT) is a factor that influences the final results, and their limitations are not fully quantified in isolation.
  • Future Work: The authors suggest exploring more sophisticated SPARQL queries for entity selection, extending the approach to other languages and domains beyond medicine, and investigating methods for handling nested or discontinuous entities. They also highlight the potential for improved performance with more advanced pre-trained models.

Summary updated

BibTeX
@inproceedings{frei2024creating,
  author = {Johann Frei and Frank Kramer},
  title = {{Creating Ontology-annotated Corpora from Wikipedia for Medical Named-entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure