Introduction
Knowledge Graphs (KGs) serve as the foundation for modern semantic search and recommendation systems. However, constructing high-quality KGs remains a significant challenge. Manual curation is slow and expensive, while automated extraction using Large Language Models (LLMs) often suffers from inconsistency, domain gaps, and a lack of interoperability with existing standards like Wikidata. This paper introduces a novel pipeline that leverages LLMs to build structured KGs from unstructured text by grounding them in the Wikidata schema.
Methodology
The authors propose a four-stage "ontology-grounded" pipeline designed to ensure both accuracy and interoperability.
- Competency Question (CQ) Generation: The LLM first generates questions and answers based on the document (e.g., "What is the occupation of X?"). This step defines the scope of knowledge to be captured.
- Relation Extraction & Ontology Matching: Potential relations are extracted from the CQs. The system uses vector similarity to find the closest matching property in the Wikidata schema, which is then vetted by an LLM for semantic accuracy.
- Ontology Formatting: A formal OWL (Web Ontology Language) file is generated. If a relation is missing from Wikidata, the LLM "mints" a new property, inferring its domain and range.
- KG Construction: Finally, the LLM transforms the CQ-answer pairs into structured RDF triples, strictly following the generated ontology.
Experimental Setup
The approach was evaluated on three benchmark datasets:
- Wiki-NRE: Derived from Wikipedia and Wikidata.
- WebNLG: Used for natural language generation and semantic parsing.
- SciERC: Contains scientific knowledge (e.g., machine learning terms), testing the system's ability to handle relations outside the standard Wikidata set.
Key Results
The proposed method, particularly when utilizing GPT-4o, significantly outperformed traditional baselines and previous LLM-based approaches across all datasets.
On the SciERC dataset, the pipeline achieved a 0.77 F1 score, a dramatic improvement over the 0.07 score of previous LLM baselines that lacked this grounded approach. For Wiki-NRE, it reached a 0.71 F1 score, surpassing the previous best LLM baseline of 0.647. On WebNLG, the zero-shot approach remained highly competitive with a 0.76 F1 score.
Discussion and Insights
- Scoping via CQs: Competency Questions act as an effective filter, forcing the LLM to identify critical information before formatting it into a graph.
- Latent Knowledge: The authors found that LLMs can spontaneously recall Wikidata-style formatting, which the pipeline leverages to improve alignment.
- The Schema Trade-off: Interestingly, performance scores sometimes drop when the system is allowed to discover any relation without schema constraints. This occurs because the LLM identifies valid information not present in the "ground truth" annotations, suggesting that current benchmarks may underestimate LLM capabilities.
- Interoperability: By utilizing Wikidata property IDs (e.g., P106 for "occupation"), the resulting KG can be immediately compatible with large-scale open knowledge bases.
Limitations and Future Work
Despite its success, the pipeline is computationally intensive, requiring multiple LLM calls per document. Future research will explore "guided decoding" to improve efficiency and few-shot fine-tuning to enhance the performance of smaller models like Mistral-7B.
Summary updated