Ontology-guided Knowledge Graph Construction from Maintenance Short Texts

Zeno van Cauter, Nikolay YakovetsACL 2024

TL;DR

This paper explores using Large Language Models (LLMs), specifically the Llama family, for constructing knowledge graphs from specialized maintenance texts. By combining ontology guidance with in-context learning, they demonstrate that LLMs can achieve performance comparable to fine-tuned models with significantly less labeled data, opening doors for more efficient domain-specific knowledge graph creation.

What Problem Does This Paper Address?

Building comprehensive knowledge graphs (KGs) is crucial for various applications, but it's often hindered by the significant human effort required for data annotation. This challenge is amplified when dealing with domain-specific data, which has unique vocabularies and contexts that off-the-shelf NLP solutions struggle with. Furthermore, the cost of annotating specialized data is prohibitive, and the typically small datasets make it difficult to train robust models. Existing methods also face issues with evolving ontologies, requiring expensive retraining. This paper tackles the problem of efficiently constructing KGs from domain-specific "Maintenance Short Texts" (MSTs) by leveraging the power of LLMs.

Methods

The core of this work lies in an ontology-guided triplet extraction approach powered by LLMs. Instead of traditional fine-tuning, the authors employ in-context learning (ICL). This means they provide the LLM with a few semantically similar examples within the prompt itself, along with an ontology and the text to be processed. The LLM then infers the desired output format and extracts facts (subject-relation-object triples) based on this context.

Key aspects of their methodology include:

  • Ontology Guidance: The LLM is given a predefined ontology (a set of concepts and relations) to constrain its output and ensure it adheres to a structured representation.
  • In-Context Learning (ICL): The LLM learns from a small number of examples provided directly in the prompt, eliminating the need for extensive model fine-tuning.
  • Token Prediction Penalties: They investigate the use of "frequency penalty" and "presence penalty" parameters to control LLM behavior, aiming to reduce repetitive token generation and encourage diversity, thereby mitigating hallucinations.
  • Hallucination Filtering: A post-processing step is introduced to filter out triples that do not conform to the ontology or contain subjects/objects not present in the original text, improving precision.

Datasets

The primary dataset used in this study is MaintIE (Maintenance Short Texts), which contains information from Maintenance Work Orders. This dataset comes in two versions:

  • Fine-grained: Spans 224 entity classes.
  • Course-grained: Spans 6 entity classes.

The authors opt for the course-grained version due to its larger size (7,000 examples vs. 1,067). They filter out texts without annotated triples, resulting in 6,728 examples, which are then split into 75/25 train-test sets. Examples for the in-context learning prompts are retrieved from the training split using sentence transformers.

Key Insights and Ideas

  • LLMs as powerful few-shot learners for domain-specific IE: The study demonstrates that LLMs, particularly instruction-tuned variants like Llama-3-70B-Instruct, can achieve impressive fact extraction performance with as few as 20 semantically similar in-context examples. This significantly reduces the reliance on large labeled datasets.
  • Inference-only approach is viable: The findings suggest that domain-specific fact extraction can be accomplished through LLM inference alone, bypassing the need for computationally expensive fine-tuning.
  • Hallucinations are a key challenge, but manageable: While LLMs are prone to generating incorrect or fabricated information (hallucinations), the paper shows that these can be mitigated through careful tuning of prediction penalties and effectively filtered out post-extraction.
  • Pruning hallucinations boosts performance: Removing hallucinated triples significantly improves precision and F1-scores, making even smaller LLMs like Llama-3-8B viable alternatives.
  • Ontology conformance is high: LLMs generally adhere well to the provided ontology, often better than in previous benchmarks, indicating their ability to follow structured guidance.

Key Results

  • Comparable Performance: Llama-3-70B-Instruct, using only 20 in-context examples, achieved an F1-score of 0.69, which is remarkably close to the state-of-the-art supervised fine-tuning method (REBEL on MaintIE) that obtained an F1-score of 0.77.
  • Impact of Instruction Tuning: For Llama-3-8B, instruction tuning slightly decreased performance, while for Llama-3-70B, it was crucial for achieving any meaningful performance.
  • Hallucination Reduction: By applying a simple filter for verifiable hallucinations, models saw significant performance gains, particularly in precision.
  • Model Differences: Stark differences were observed between Llama-3-70B models, with the instruction-tuned version outperforming the base model significantly. Llama-3-8B-Instruct showed a lower ontology conformance compared to other models.

Limitations and Future Work

  • Hallucination Analysis Complexity: The authors acknowledge that a comprehensive framework for analyzing and proving causality for hallucinations is complex and beyond the scope of this work, suggesting it as a direction for future research.
  • Ontology Size and Complexity: The current approach is limited by the context window of LLMs, which can become a bottleneck for larger or more complex ontologies. This is identified as an avenue for future work.
  • Extended Evaluation Framework: The paper suggests that an extended evaluation framework for understanding the nuances of hallucinations would be beneficial.
  • Human-in-the-Loop Integration: The work implies that LLMs can facilitate human-in-the-loop pre-annotation, reducing the workload for domain experts. Further research could explore optimizing this interactive process.

Summary updated

BibTeX
@inproceedings{cauter2024ontology,
  author = {Zeno van Cauter and Nikolay Yakovets},
  title = {{Ontology-guided Knowledge Graph Construction from Maintenance Short Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure