Enriching Ontologies with Disjointness Axioms using Large Language Models

Elias Crum, Antonio De Santis, Manon Ovide, Jiaxin Pan, Alessia Pisu, Nicolas Lazzari, Sebastian Rudolph

Abstract

Ontologies often lack explicit disjointness declarations between classes, despite their usefulness for sophisticated reasoning and consistency checking in Knowledge Graphs. In this study, we explore the potential of Large Language Models (LLMs) to enrich ontologies by identifying and asserting class disjointness axioms. Our approach aims at leveraging the implicit knowledge embedded in LLMs, using prompt engineering to elicit this knowledge for classifying ontological disjointness. We validate our methodology on the DBpedia ontology, focusing on open-source LLMs. Our findings suggest that LLMs, when guided by effective prompt strategies, can reliably identify disjoint class relationships, thus streamlining the process of ontology completion without extensive manual input. For comprehensive disjointness enrichment, we propose a process that takes logical relationships between disjointness and subclass statements into account in order to maintain satisfiability and reduce the number of calls to the LLM.

Introduction

In the development of Knowledge Graphs (KGs), significant effort is spent on defining class memberships (e.g., "Flipper is a Mammal"). However, the inverse—defining what an entity cannot be—is frequently overlooked. These "disjointness axioms" (e.g., "A Mammal cannot be a Fish") are essential for automated reasoning and maintaining graph consistency. Because human experts often find these rules intuitive, they are rarely explicitly documented, making manual enrichment of large ontologies like DBpedia a daunting task. This research investigates whether the "common sense" knowledge within Large Language Models (LLMs) can be harnessed to automatically identify these missing axioms.

Methodology: A Hybrid Logic-LLM Approach

The proposed methodology combines the linguistic intuition of LLMs with the formal rigor of ontological reasoning. The process involves three main stages:

1. Logical Pre-processing and Propagation

Before querying an LLM, the system applies formal logic to propagate known disjointness. For instance, if "Person" and "Place" are disjoint, any subclass of "Person" is logically disjoint from any subclass of "Place." This step significantly reduces the number of pairs that require LLM evaluation.

Examples of class pairs and the logical propositions used to determine their disjointness status.

2. Strategic Prompting

The authors evaluated several prompting strategies to elicit the most accurate responses from the LLMs:

  • Negative QA: Framing the question as a possibility—"Can a [Class A] be a [Class B]?"—consistently outperformed direct questions about disjointness. This is likely because LLMs are better at determining linguistic possibility than formal ontological properties.
  • Zero-shot vs. Few-shot: Interestingly, zero-shot prompting (providing only a task description) often yielded better results than few-shot prompting. This suggests that for general "world knowledge" tasks, specific examples might introduce unwanted bias.

3. Logical Pruning

After the LLM provides its classifications, a reasoner is used to remove redundant statements. This ensures the final set of axioms is minimal and logically sound.

Evaluation and Datasets

The approach was validated using the DBpedia ontology, a massive crowdsourced knowledge base. The researchers extracted 1,148 class pairs, including a ground truth set of 370 disjoint and 778 non-disjoint pairs to serve as a benchmark.

Sample class pairs from the DBpedia ontology used for evaluating the LLM's classification performance.

Key Results and Insights

  • Performance: Gemma 2 9B was identified as the top-performing model, particularly when utilizing the "Negative QA" strategy. Mistral 0.3 7B also showed high recall, making it effective for identifying a broad range of potential disjointness axioms.
  • Efficiency: The hybrid approach identified over 510,000 axioms across DBpedia. The logical pruning step successfully reduced this set by 66%, resulting in a manageable core of approximately 170,000 axioms without any loss of information.
  • Contextual Challenges: While successful, the models still struggle with context-dependent classes. For example, a "Mosque" is primarily a "Place" but may also function as a "Museum" (Organization) in specific historical or cultural contexts.

Conclusion

This work demonstrates that LLMs, when integrated with formal logical frameworks, provide a powerful tool for automated ontology enrichment. By leveraging "Negative QA" prompting and logical propagation, researchers can significantly reduce the manual effort required to maintain consistent and robust Knowledge Graphs. Future work will explore Retrieval-Augmented Generation (RAG) to provide models with more specific context, further improving accuracy in complex domains.

Code available at: https://github.com/n28div/llm-disjointness

Summary updated

Abstract

Ontologies often lack explicit disjointness declarations between classes, despite their usefulness for sophisticated reasoning and consistency checking in Knowledge Graphs. In this study, we explore the potential of Large Language Models (LLMs) to enrich ontologies by identifying and asserting class disjointness axioms. Our approach aims at leveraging the implicit knowledge embedded in LLMs, using prompt engineering to elicit this knowledge for classifying ontological disjointness. We validate our methodology on the DBpedia ontology, focusing on open-source LLMs. Our findings suggest that LLMs, when guided by effective prompt strategies, can reliably identify disjoint class relationships, thus streamlining the process of ontology completion without extensive manual input. For comprehensive disjointness enrichment, we propose a process that takes logical relationships between disjointness and subclass statements into account in order to maintain satisfiability and reduce the number of calls to the LLM. This work provides a foundation for future applications of LLMs in automated ontology enhancement and offers insights into optimizing LLM performance through strategic prompt design. Our code is publicly available on GitHub at https://github.com/n28div/llm-disjointness.
BibTeX
@inproceedings{crum2024enriching,
  author = {Elias Crum and Antonio De Santis and Manon Ovide and Jiaxin Pan and Alessia Pisu and Nicolas Lazzari and Sebastian Rudolph},
  title = {{Enriching Ontologies with Disjointness Axioms using Large Language Models}},
  booktitle = {KBC-LM/LM-KBC@ISWC},
  year = {2024},
  url = {https://openreview.net/forum?id=k2W07EBum9},
}
Figure