Abstract
The task of building semantics for structured data such as CSV, JSON, and XML files is highly relevant in the knowledge representation field. Even though we have a vast amount of structured data on the internet, mapping them to domain ontologies to build semantics for them is still very challenging as it requires the construction model to understand and learn graph-structured knowledge. Otherwise, the task will require significant human effort and cost. In this paper, we propose a novel automatic semantic modeling framework: Knowledge Prompt Chaining (KPC). It serializes graph-structured knowledge and injects it into Large Language Models (LLMs) using a prompt chaining architecture. Through this knowledge injection and prompt chaining, the model learns the structural information and latent space of the graph, generating semantic labels and semantic graphs following the chain's instructions naturally. Based on experimental results, our method achieves better performance than existing leading techniques, despite using reduced structured input data.
Introduction
The proliferation of structured data across the internet—in formats like CSV, XML, and JSON—presents a significant opportunity for data integration. However, much of this data remains siloed due to varying naming conventions and schemas. To unlock its full potential for AI applications, this data must be mapped to formal domain ontologies—a process known as Semantic Modeling.
Traditionally, semantic modeling has been a labor-intensive task, requiring either manual expert intervention or complex algorithmic approaches like Steiner trees and Graph Neural Networks. These traditional methods often struggle with ontology reasoning, such as understanding hierarchical relationships between classes. This paper introduces Knowledge Prompt Chaining (KPC), a framework that leverages the inherent reasoning capabilities of LLMs to automate this process efficiently.
Methodology: Knowledge Prompt Chaining
The KPC framework addresses the challenges of semantic modeling through three core pillars: knowledge serialization, prompt chaining, and knowledge injection with pruning.
1. Knowledge Serialization
Because LLMs are optimized for text, the framework converts structured data and graph-based ontologies into a unified JSON format. A key finding of this research is that providing as few as three rows of data is sufficient for the model to grasp the context, which significantly reduces token consumption and computational cost.
2. Prompt Chaining
Rather than attempting to generate a complete semantic model in a single pass, KPC decomposes the task into a two-step chain:
- Step 1: Semantic Labeling (Chain 1): The LLM identifies the semantic type for each attribute in the input data (e.g., mapping a column to an "Artist Name" concept).
- Step 2: Semantic Graph Building (Chain 2): Using the labels from the first step, the model identifies the relationships between these entities (e.g., establishing that an "Artist" created an "Artwork").
3. Knowledge Injection and Pruning
The framework utilizes "Long In-context Learning" (LongICL) to feed the entire domain ontology into the system prompt. To mitigate hallucinations—where the LLM might propose non-existent relationships—a pruning step is implemented. This step removes any generated nodes that cannot be logically connected back to the actual data attributes based on the ontology's rules.
Experimental Evaluation
The KPC framework was evaluated using three real-world datasets: $ds_{crm}$ (museum data mapped to CIDOC CRM), $ds_{edm}$ (museum data using the Europeana Data Model), and $ds_{schema}$ (football data using Schema.org extensions). The researchers updated the "gold standard" benchmarks for the museum datasets to ensure they were logical and interpretable for modern modeling standards.
Key Insights
- Efficiency via Sparsity: KPC achieved peak performance using only a tiny fraction of the available data points, outperforming methods that require processing entire tables.
- Reasoning over Rules: Requiring the LLM to provide a "detailed reasoning process" (Chain-of-Thought) significantly improved the reliability of semantic links.
- The "Overthinking" Phenomenon: Interestingly, advanced models like Claude 3.5 and GPT-4 occasionally performed worse on very simple graphs, suggesting they may look for complex relationships where none exist.
Results
The KPC framework demonstrated superior performance compared to existing state-of-the-art techniques like Serene and PGM-SM:
- High Accuracy: On the $ds_{edm}$ dataset, the framework achieved 90.4% precision and 92.9% recall.
- Labeling Precision: Semantic labeling accuracy ranged from 96.9% to 99.1% across datasets when using Claude 3.5 Sonnet.
- Model Performance: Claude 3.5 Sonnet was the top performer, followed by DeepSeek-V2.5 and GPT-4 Turbo.
- Operational Speed: The average prediction time per data source was under 30 seconds, making the framework suitable for real-time integration.
Conclusion and Future Work
Knowledge Prompt Chaining significantly reduces the human effort required for semantic modeling. While it currently relies on some manually defined rules to prevent hallucinations, future work will explore Meta-Prompting. This would involve the LLM "teaching itself" the rules by analyzing the ontology, moving toward a fully autonomous system. Additionally, the reasoning chains generated by the LLM could serve as synthetic data for fine-tuning smaller, specialized models.
Summary updated