Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema

Xiaohan Feng, Xixin Wu, Helen Meng

Abstract

We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base. An ontology is authored by generating Competency Questions (CQ) on a knowledge base to discover knowledge scope, extracting relations from CQs, and attempting to replace equivalent relations with their counterparts in Wikidata. To ensure consistency and interpretability in the resulting KG, we ground the generation of the KG with the authored ontology based on extracted relations. Evaluation on benchmark datasets demonstrates competitive performance in the knowledge graph construction task. Our work presents a promising direction for a scalable KG construction pipeline with minimal human intervention that yields high-quality and human-interpretable KGs, which are interoperable with Wikidata semantics for potential knowledge base expansion.

Introduction

Knowledge Graphs (KGs) are fundamental to modern semantic search and reasoning systems. However, their construction has traditionally been a manual, labor-intensive process. While Large Language Models (LLMs) offer a potential shortcut by extracting facts directly from unstructured text, they often struggle with three critical issues: inconsistency (redundant or conflicting relations), domain bias (relying on pre-training data over the provided document), and a lack of interoperability with existing standards like Wikidata.

Methodology

The proposed pipeline addresses these challenges through a four-stage "ontology-grounded" process designed to enforce structural and semantic consistency.

Overview of the ontology-grounded KG construction pipeline using Douglas Adams as an example.
  1. Competency Question (CQ) Generation: The LLM first analyzes the source document to generate questions it can answer (e.g., "What is the occupation of Douglas Adams?"). This step defines the scope of knowledge before extraction begins.
  2. Relation Extraction & Ontology Matching: Potential relations are extracted from these questions and matched against the Wikidata property list using vector similarity (via bge-small-en embeddings). A secondary LLM pass vets these matches to ensure semantic accuracy.
  3. Ontology Formatting: The system generates a formal OWL (Web Ontology Language) file. If a relation is missing from Wikidata, the LLM "mints" a new property, inferring its domain and range.
  4. Grounded KG Construction: Finally, the LLM transforms the document's facts into RDF triples, strictly constrained to use only the properties defined in the newly authored ontology.

Evaluation and Results

The approach was evaluated on three benchmark datasets: Wiki-NRE (Wikipedia/Wikidata derived), SciERC (scientific entities), and WebNLG (natural language to structured data mapping).

Performance comparison (F1 scores) across benchmark datasets.

As shown in the results table, the proposed method using GPT-4o achieved superior performance on Wiki-NRE and SciERC compared to both non-LLM and standard LLM baselines. A particularly notable breakthrough occurred in the scientific domain (SciERC), where a direct LLM baseline scored a dismal 0.07 F1, while the grounded approach reached 0.77 F1. This highlights the necessity of a grounding schema for specialized domains.

Discussion and Insights

  • Questions as a Bridge: Using Competency Questions acts as an automated "requirements engineering" phase, forcing the LLM to establish a schema before data entry.
  • The Wikidata Anchor: By aligning with Wikidata, the resulting KG is not an isolated silo but a potential extension of the global open knowledge base.
  • The "Richness" Penalty: Interestingly, unconstrained LLMs sometimes scored lower on F1 metrics not due to errors, but because they extracted valid facts that human annotators had missed in the original benchmarks.

Limitations and Future Work

Despite its success, the pipeline is computationally expensive due to multiple LLM calls per document. Additionally, as ontologies grow, fitting the entire schema into a prompt can lead to the "lost in the middle" phenomenon. Future research will explore "guided decoding" to force output grammar and few-shot fine-tuning to improve efficiency and reduce costs.

Summary updated

Abstract

We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base. An ontology is authored by generating Competency Questions (CQ) on knowledge base to discover knowledge scope, extracting relations from CQs, and attempt to replace equivalent relations by their counterpart in Wikidata. To ensure consistency and interpretability in the resulting KG, we ground generation of KG with the authored ontology based on extracted relations. Evaluation on benchmark datasets demonstrates competitive performance in knowledge graph construction task. Our work presents a promising direction for scalable KG construction pipeline with minimal human intervention, that yields high quality and human-interpretable KGs, which are interoperable with Wikidata semantics for potential knowledge base expansion.
BibTeX
@inproceedings{feng2024ontology,
  author = {Xiaohan Feng and Xixin Wu and Helen Meng},
  title = {{Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema}},
  booktitle = {HI-AI @ KDD},
  year = {2024},
  url = {https://openreview.net/forum?id=pOg4iFtSdi},
}
Figure