Introduction
In modern corporate environments, critical knowledge is often siloed within unstructured documents such as internal wikis, HR manuals, and technical reports. To leverage this information for advanced AI applications like question-answering or recommendation engines, organizations must transform this text into structured Knowledge Bases (KBs). Researchers at Orange have developed a production-ready pipeline designed to extract knowledge triples (subject-relation-object) using the standardized Wikidata schema.
The Challenge of Industrial Knowledge Extraction
The primary hurdle in "Knowledge Extraction" (KE) is accurately identifying entities and the specific relationships between them in "wild" data. While academic models excel on curated Wikipedia-style text, they frequently struggle with industrial datasets characterized by specialized jargon, overlapping information, and less rigid structure.
A Modular Extraction Pipeline
The proposed system utilizes a classical NLP pipeline consisting of four distinct stages:
- Sentence Splitting: Utilizing
pySBDto segment documents into manageable units. - Entity Detection and Linking: Implementing GENRE to identify and link names and common nouns to unique Wikidata identifiers.
- Coreference Resolution: Employing
neuralcorefto resolve pronouns (e.g., "it") back to their original entities. - Relation Extraction (RE): The core of the pipeline, featuring a fine-tuned RoBERTa-large model trained to classify relationships between entity pairs.
Data Engineering and Unified Datasets
A significant contribution of this work is the creation of a massive, unified relation extraction database. By mapping six major existing datasets to the Wikidata schema, the researchers created a consolidated resource for training and experimentation.
This database contains over 47 million instances across 1,022 unique Wikidata relation types, providing a robust foundation for training models on a wide variety of semantic connections.
Key Insights and Performance Metrics
The research highlights several critical findings for deploying KE systems in real-world scenarios:
- The "Balanced" Advantage: The study found that data quality and balance are more important than sheer volume. A "WebRED-balanced" model, which included manually added examples for underrepresented relations, significantly outperformed models trained on larger but imbalanced datasets.
- The Reality Gap: Moving from academic benchmarks to internal corporate data incurs a "performance tax." While entity linking remained relatively strong (F1 of 0.71), the Relation Extraction model faced difficulties with the domain shift.
- The Necessity of Negative Examples: In real-world text, most entity pairs are unrelated. Including a "no relation" (P0) class during training is vital for minimizing false positives in production.
As shown in the performance table, the balanced training approach yielded the highest F1 scores, demonstrating the effectiveness of targeted data engineering.
Conclusion and Future Directions
While the pipeline demonstrates strong performance on benchmarks, the "notoriously difficult" task of relation extraction in specialized domains remains a challenge. Future work will focus on integrating syntactic and semantic parsing to better capture complex sentence structures and employing active learning to fine-tune models on domain-specific corporate data. By using Wikidata as a "lingua franca," the system ensures interoperability with a vast ecosystem of public knowledge.
Summary updated