Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema

Xiaohan Feng, Xixin Wu, Helen Meng

Introduction

Knowledge Graphs (KGs) serve as the foundation for modern semantic search and recommendation systems. However, constructing high-quality KGs remains a significant challenge. Manual curation is slow and expensive, while automated extraction using Large Language Models (LLMs) often suffers from inconsistency, domain gaps, and a lack of interoperability with existing standards like Wikidata. This paper introduces a novel pipeline that leverages LLMs to build structured KGs from unstructured text by grounding them in the Wikidata schema.

Methodology

The authors propose a four-stage "ontology-grounded" pipeline designed to ensure both accuracy and interoperability.

The four-stage pipeline for ontology-grounded KG construction, moving from document text to structured RDF triples.
  1. Competency Question (CQ) Generation: The LLM first generates questions and answers based on the document (e.g., "What is the occupation of X?"). This step defines the scope of knowledge to be captured.
  2. Relation Extraction & Ontology Matching: Potential relations are extracted from the CQs. The system uses vector similarity to find the closest matching property in the Wikidata schema, which is then vetted by an LLM for semantic accuracy.
  3. Ontology Formatting: A formal OWL (Web Ontology Language) file is generated. If a relation is missing from Wikidata, the LLM "mints" a new property, inferring its domain and range.
  4. KG Construction: Finally, the LLM transforms the CQ-answer pairs into structured RDF triples, strictly following the generated ontology.

Experimental Setup

The approach was evaluated on three benchmark datasets:

  • Wiki-NRE: Derived from Wikipedia and Wikidata.
  • WebNLG: Used for natural language generation and semantic parsing.
  • SciERC: Contains scientific knowledge (e.g., machine learning terms), testing the system's ability to handle relations outside the standard Wikidata set.

Key Results

The proposed method, particularly when utilizing GPT-4o, significantly outperformed traditional baselines and previous LLM-based approaches across all datasets.

Performance comparison (F1 scores) of the proposed method against baselines on Wiki-NRE, SciERC, and WebNLG datasets.

On the SciERC dataset, the pipeline achieved a 0.77 F1 score, a dramatic improvement over the 0.07 score of previous LLM baselines that lacked this grounded approach. For Wiki-NRE, it reached a 0.71 F1 score, surpassing the previous best LLM baseline of 0.647. On WebNLG, the zero-shot approach remained highly competitive with a 0.76 F1 score.

Discussion and Insights

  • Scoping via CQs: Competency Questions act as an effective filter, forcing the LLM to identify critical information before formatting it into a graph.
  • Latent Knowledge: The authors found that LLMs can spontaneously recall Wikidata-style formatting, which the pipeline leverages to improve alignment.
  • The Schema Trade-off: Interestingly, performance scores sometimes drop when the system is allowed to discover any relation without schema constraints. This occurs because the LLM identifies valid information not present in the "ground truth" annotations, suggesting that current benchmarks may underestimate LLM capabilities.
  • Interoperability: By utilizing Wikidata property IDs (e.g., P106 for "occupation"), the resulting KG can be immediately compatible with large-scale open knowledge bases.

Limitations and Future Work

Despite its success, the pipeline is computationally intensive, requiring multiple LLM calls per document. Future research will explore "guided decoding" to improve efficiency and few-shot fine-tuning to enhance the performance of smaller models like Mistral-7B.

Summary updated

Abstract

We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base. An ontology is authored by generating Competency Questions (CQ) on knowledge base to discover knowledge scope, extracting relations from CQs, and attempt to replace equivalent relations by their counterpart in Wikidata. To ensure consistency and interpretability in the resulting KG, we ground generation of KG with the authored ontology based on extracted relations. Evaluation on benchmark datasets demonstrates competitive performance in knowledge graph construction task. Our work presents a promising direction for scalable KG construction pipeline with minimal human intervention, that yields high quality and human-interpretable KGs, which are interoperable with Wikidata semantics for potential knowledge base expansion.
BibTeX
@inproceedings{feng2024ontology,
  author = {Xiaohan Feng and Xixin Wu and Helen Meng},
  title = {{Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema}},
  booktitle = {HI-AI @ KDD},
  year = {2024},
  url = {https://openreview.net/forum?id=pOg4iFtSdi},
}
Figure