Knowledge Prompting: How Knowledge Engineers Use Large Language Models

Elisavet Koutsiana, Johanna Walker, Michelle Nwachukwu, Albert Meroño-Peñuela, Elena Simperl

Introduction

Knowledge Engineering (KE) is the foundational discipline of representing human expertise in machine-readable formats, most notably Knowledge Graphs (KGs). Traditionally, this has been a labor-intensive process requiring deep domain expertise and manual curation. The emergence of Large Language Models (LLMs) like GPT-4 has introduced a paradigm shift, promising to automate complex tasks but also introducing new challenges regarding reliability, bias, and transparency.

This research investigates how professional knowledge engineers are integrating these "black box" tools into their workflows. By moving from manual triple-crafting to what the authors term "knowledge prompting," the community faces a critical evaluation gap that necessitates new documentation frameworks like KG Cards.

The Knowledge Graph Lifecycle

To understand where LLMs fit, it is essential to look at the standard lifecycle of a Knowledge Graph, which spans from initial requirement collection to deployment and maintenance.

The Knowledge Graph Lifecycle: Stages from requirement collection to deployment.

As shown in the lifecycle above, LLMs can potentially impact every stage—from extracting entities during construction (Stage B) to assisting in reasoning and refinement during maintenance (Stage C).

Methodology: The "LLMs in KE" Hackathon

The researchers conducted a qualitative study centered on a four-day hackathon involving 39 participants, including PhD students and industry practitioners. The study employed a multi-method approach:

  • Ethnographic Observation: Researchers took detailed notes on group decision-making and prompting strategies.
  • Artifact Analysis: Review of 7 hackathon reports and presentations.
  • Interviews: 14 semi-structured follow-up interviews focusing on trust, safety, and required skills.

The hackathon covered a wide range of topics, testing the limits of LLMs across various KE tasks, from ontology alignment to multimodal graph construction.

Hackathon topics and the corresponding Knowledge Engineering and GenAI tasks.

Key Insights: From Building to Auditing

The study revealed several critical shifts in the KE profession:

1. The Rise of "Knowledge Prompting"

Prompting is no longer seen as a simple interaction but as a technical skill requiring domain expertise. Effective "Knowledge Prompting" involves providing the LLM with specific context and personas, making it a core competency comparable to writing SPARQL or OWL.

2. The Evaluation Crisis

While LLMs can generate data rapidly, verifying that data remains a bottleneck. Standard metrics like F1-scores are often inadequate for complex ontologies without a "gold standard." Participants frequently relied on manual "eyeballing," which is neither scalable nor rigorous.

3. The Bias Fallacy

A significant finding was the tendency for engineers to view AI safety and bias as external issues for ethics researchers rather than an integral part of the engineering process. There was a notable lack of practical strategies for mitigating bias within the generated graphs.

Proposed Solution: KG Cards

To address the transparency gap, the authors propose KG Cards, inspired by Model and Data Cards. These structured documents track a graph's provenance, construction methods (including specific LLM usage), and quality metrics. This framework aims to build trust by making the "black box" elements of AI-assisted construction visible to end-users across six dimensions: Knowledge, Provenance, Graph Construction, Quality, Licensing, and Transparency.

Conclusion

The role of the Knowledge Engineer is evolving from a "builder from scratch" to a "curator and auditor" of AI-generated knowledge. While LLMs offer significant efficiency gains in the initial stages of graph construction, the time saved is often redirected toward rigorous human-in-the-loop verification. Future work will focus on developing automated evaluation toolkits to judge the logical soundness of LLM outputs and formalizing the KG Card standard within the Semantic Web community.

Summary updated

Abstract

Despite many advances in knowledge engineering (KE), challenges remain in areas such as engineering knowledge graphs (KGs) at scale, keeping up with evolving domain knowledge, multilingualism, and multimodality. Recently, KE has used LLMs to support semi-automatic tasks, but the most effective use of LLMs to support knowledge engineers across the KE activites is still in its infancy. To explore the vision of LLM copilots for KE and change existing KE practices, we conducted a multimethod study during a KE hackathon. We investigated participants' views on the use of LLMs, the challenges they face, the skills they may need to integrate LLMs into their practices, and how they use LLMs responsibly. We found participants felt LLMs could contribute to improving efficiency when engineering KGs, but presented increased challenges around the already complex issues of evaluating the KE tasks. We discovered prompting to be a useful but undervalued skill for knowledge engineers working with LLMs, and note that natural language processing skills may become more relevant across more roles in KG construction. Integrating LLMs into KE tasks needs to be mindful of potential risks and harms related to responsible AI. Given the limited ethical training, most knowledge engineers receive solutions such as our suggested `KG cards' based on data cards could be a useful guide for KG construction. Our findings can support designers of KE AI copilots, KE researchers, and practitioners using advanced AI to develop trustworthy applications, propose new methodologies for KE and operate new technologies responsibly.
BibTeX
@article{koutsiana2026knowledge,
  author = {Elisavet Koutsiana and Johanna Walker and Michelle Nwachukwu and Albert Meroño-Peñuela and Elena Simperl},
  title = {{Knowledge Prompting: How Knowledge Engineers Use Large Language Models}},
  journal = {J. Web Semant.},
  year = {2026},
  url = {https://openreview.net/forum?id=RQWKYT7O3a},
}
Figure