Knowledge Extraction From Texts Based on Wikidata

Anastasia Shimorina, Johannes Heinecke, Frédéric HerledanNAACL 2022 · DOI

Introduction

In modern corporate environments, critical knowledge is often siloed within unstructured documents such as internal wikis, HR manuals, and technical reports. To leverage this information for advanced AI applications like question-answering or recommendation engines, organizations must transform this text into structured Knowledge Bases (KBs). Researchers at Orange have developed a production-ready pipeline designed to extract knowledge triples (subject-relation-object) using the standardized Wikidata schema.

The Challenge of Industrial Knowledge Extraction

The primary hurdle in "Knowledge Extraction" (KE) is accurately identifying entities and the specific relationships between them in "wild" data. While academic models excel on curated Wikipedia-style text, they frequently struggle with industrial datasets characterized by specialized jargon, overlapping information, and less rigid structure.

A Modular Extraction Pipeline

The proposed system utilizes a classical NLP pipeline consisting of four distinct stages:

  1. Sentence Splitting: Utilizing pySBD to segment documents into manageable units.
  2. Entity Detection and Linking: Implementing GENRE to identify and link names and common nouns to unique Wikidata identifiers.
  3. Coreference Resolution: Employing neuralcoref to resolve pronouns (e.g., "it") back to their original entities.
  4. Relation Extraction (RE): The core of the pipeline, featuring a fine-tuned RoBERTa-large model trained to classify relationships between entity pairs.

Data Engineering and Unified Datasets

A significant contribution of this work is the creation of a massive, unified relation extraction database. By mapping six major existing datasets to the Wikidata schema, the researchers created a consolidated resource for training and experimentation.

Comparison of the unified database (DB) with existing relation extraction datasets.

This database contains over 47 million instances across 1,022 unique Wikidata relation types, providing a robust foundation for training models on a wide variety of semantic connections.

Key Insights and Performance Metrics

The research highlights several critical findings for deploying KE systems in real-world scenarios:

  • The "Balanced" Advantage: The study found that data quality and balance are more important than sheer volume. A "WebRED-balanced" model, which included manually added examples for underrepresented relations, significantly outperformed models trained on larger but imbalanced datasets.
  • The Reality Gap: Moving from academic benchmarks to internal corporate data incurs a "performance tax." While entity linking remained relatively strong (F1 of 0.71), the Relation Extraction model faced difficulties with the domain shift.
  • The Necessity of Negative Examples: In real-world text, most entity pairs are unrelated. Including a "no relation" (P0) class during training is vital for minimizing false positives in production.
Performance metrics for different training scenarios on the WebRED validation set.

As shown in the performance table, the balanced training approach yielded the highest F1 scores, demonstrating the effectiveness of targeted data engineering.

Conclusion and Future Directions

While the pipeline demonstrates strong performance on benchmarks, the "notoriously difficult" task of relation extraction in specialized domains remains a challenge. Future work will focus on integrating syntactic and semantic parsing to better capture complex sentence structures and employing active learning to fine-tune models on domain-specific corporate data. By using Wikidata as a "lingua franca," the system ensures interoperability with a vast ecosystem of public knowledge.

Summary updated

BibTeX
@inproceedings{shimorina2022knowledge,
  author = {Anastasia Shimorina and Johannes Heinecke and Frédéric Herledan},
  title = {{Knowledge Extraction From Texts Based on Wikidata}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-industry.33},
}
Figure